Patent ID: 11936673
Assignee: GROUP IB, LTD
Field: Computer technology (Electrical engineering)
Classification: CPC G  H | IPC G  H

Claim 16:
17. A system for detecting harmful content on a network, the system comprising a computing device, the computing device further comprising:
at least one processor;
at least one non-transitory computer-readable medium comprising instructions, which, when executed by the at least one processor, cause the system to:
receive, via the network, a URL;
obtain, from the URL, an HTML document associated therewith;
convert the HTML document into a text by extracting a respective text portion from an HTML body of the HTML document;
normalize the text associated with the HTML document by transforming each word of the text to a respective canonic form thereof, thereby generating a plurality of tokens associated with the text;
aggregate each one of the plurality of tokens into a token vector associated with the HTML document;
apply one or more classifiers to the token vector associated with the HTML document to determine a likelihood parameter indicative of the HTML document containing the harmful content,
the one or more classifiers having been trained to determine the harmful content based on a respective training set of data, the respective training set of data having been generated by:
receiving a plurality of training HTML documents;
converting each one of the plurality of training HTML documents, into a respective training text of a plurality of training texts, the converting comprising extracting the respective text portion from an HTML body of each one of the plurality of training HTML document;
normalizing a given one of the plurality of training texts, the normalizing comprising transforming each word of the given one of the plurality of training texts to a respective canonic form thereof, thereby generating a respective plurality of training tokens associated with the given one of the plurality of training texts,
determining for each one of the respective plurality of training tokens, a respective significance parameter,  a given significance parameter associated with a given one of the plurality of training tokens being determined based on a product of multiplication of:  a local frequency of occurrence of the given one of the plurality of training tokens within the given one of the plurality training texts associated with a respective one of the plurality of training HTML documents; and  a global frequency of occurrence of the given one of the respective plurality of training tokens within the plurality of training texts of the plurality of training HTML documents;
aggregating respective pluralities of training tokens of the plurality of training HTML documents associated with respective significance parameters into a training token matrix of the respective training set of data, wherein:  a given row of the training token matrix being representative of a respective one of the plurality of training HTML documents;  a given column of the training token matrix being representative of a respective training token; and  a given element of the training token matrix being representative of a respective significance parameter of the respective token in the respective one of the plurality of training HTML documents; and

in response to the likelihood parameter being equal to or greater than a predetermined likelihood parameter threshold:
determining, by the processor, that the HTML document located under the URL contains the harmful content; and
store the URL in a database of harmful URLs.