## 5 Filtering and Searching for Information
### Searching for Information on the Internet
#### 5.1.1 Social Media Monitoring  
Within the project, authors decided to monitor the Mastodon platform infosec.exchange server [52]. The choice was made due to the growing popularity of this community and the availability of the API, which meets the requirements for conducting such monitoring and statistics. Thanks to the relatively large query limit, it is possible to download all new posts published on the server and filter them according to one's own criteria and filters. This significantly expands the possibilities and accuracy of the solution, which is not limited only to simple filtering offered by the API, such as the presence of a single keyword. This approach also reduces the need to send various queries referring to subsequent criteria. The filtered data are then indexed and stored in the Elasticsearch database, which provides many analytical mechanisms that allow for the analysis and preparation of statistics [53]. In the project, the procedure of downloading, filtering, and then indexing posts is run cyclically every 15 min. The ID of the last downloaded post is read from the Elasticsearch database, and, based on this ID, the download of all newer posts from the server is started. The API limits allow for sending a specified number of queries in a certain period of time. When the solution uses the entire pool of queries, the token renewal time is read and the downloaded posts are filtered and indexed. When a new pool of queries appears, the download is resumed and lasts until the publication time of the downloaded post exceeds the time of the program startup. The process ends with the filtering and indexing of posts. After the procedure is completed, the solution waits for the start signal. It was decided to filter posts based on an alternative selection of several conditions. In addition to deciding whether a post will be indexed or not, a \"score\" is assigned to it. This parameter allows for a preliminary determination of how much interesting information a given post contains. When calculating the score, the content, tags, and author of the entry are taken into account. The user has two lists of regular expressions (regexes) at their disposal, which allows for a more precise calculation of the final score. For each occurrence of a sequence of characters from the primary and secondary lists, 5 and 2.5 points are awarded, respectively. The user also has the option of defining their own list of users and tags, for which points are awarded. The criteria discussed in this paragraph, along with their weight, are listed below:  
* Weight of presence of a CVE identifier in the post--\(W_{\text{cve}}\);
* Weight of presence of a primary user-specified sequence of characters using regexes in the post--\(W_{\text{pc}}\);  
Figure 12: Schematic diagram of the solution for searching for information on the Internet. * Weight of presence of a secondary user-specified sequence of characters using regexes in the post--\(W_{\text{sc}}\);
* Weight of presence of a user-specified sequence of characters using regexes in the author's account description--\(W_{\text{ad}}\);
* Weight of post that was published by a user on the user-specified list of users--\(W_{\text{user}}\);
* Weight of presence of the tags on the user-specified tag list in the post--\(W_{\text{tag}}\). The way of calculating the \"score\" is defined below. Let:  
* \(p\) be the post. * \(C(p)\) be the content of the post. * \(D(p)\) be the description of the account associated with the post. * \(T(p)\) be the tags associated with the post. * \(U(p)\) be the account user. * \(r_{i}\) be a regex pattern from a list of regexes \(R\). * occurrences \((r_{i},C(p))\) be the count of regex \(r_{i}\) in content \(C(p)\). * occurrences \((r_{i},D(p))\) be the count of regex \(r_{i}\) in description \(D(p)\). * contains_user \((U(p))\) returns \(1\) if \(U(p)\) matches a user list, \(0\) otherwise. * contains_tags \((T(p))\) returns the number of matching tags between \(T(p)\) and a tag list.