# Blockchain-Based Platform to Fight Disinformation Using Crowd Wisdom and Artificial Intelligence
## 5 Use-Case Validation  
### Configuration of the Article Extraction Components  
The Offchain Core is configured to handle seven news websites. For each website specific information is required in order to perform the URL and article extraction. This information is stored in JSON format with the following properties:  
* name--news website name, which must be unique,
* urlBase--website base URL address,
* logoUrl--URL of the news outlet's logo,
* containsList--array with strings that are present in article web pages,
- containsNotList--array with strings that must not be present in order to identify a page to be an article,
* removeElements--contents that must be removed from the extracted data,
- title--article title,
- contents--article contents,
- featuredImage--URL location of the main image associated with the article,
- publishDate--publish date as it appears on the article web page (if any),
- author--author name or URL as it appears on the article web page (if any).  
\begin{table}
\begin{tabular}{l l l} \hline
**URL Path** & **Method** & **Description** \\ \hline /api/public/articles & GET & Returns a pigmented and filtered list of articles. \\  & & Request payload: it allows the following query params: pageNumber, pageSize, publisher, start date, end date, title, author, and order by date. \\  & & Response payload: [ id: number, title: string, lastUpdated: string, extractedDate: string, publishDate: string, publisher: string, author: string, url: string, contentsHash: string,...] \\ /api/public/articles/ GET & Returns information regarding the requested article-id. \\ [article-id] & Request payload: - \\  & Response payload: single article information similar to the previous web service. \\ /api/public/articles/ GET & Returns the featured image. \\ /article-id]/ & Request payload: - \\ /api/public/articles/ GET & Response payload: byte array containing the featured image. \\ [article-id]/ & Returns the article contents with the HTML tags stripped. \\ contents & Request payload: - \\ /api/public/articles/ GET & Response payload: Text representing the article contents. \\ /article-id]/ & Returns the article contents, including the HTML tags. \\ contents-full & Response payload: - \\ \hline \end{tabular}
\end{table}
Table 4: Public API routes.  
Both the crawler and the scraper are configured with the credentials to access the Crawler API and Scraper API, respectively, from the Offchain Core. Other configurations include access to the local database for each crawler/scraper and logging configurations.  
In terms of actual gathering information from the news websites, the following parameters and corresponding values were used for each crawler/scraper instance:  
* minimum waiting time between crawls from the same site (in milliseconds): 1800,
* maximum waiting time between crawls from the same site (in milliseconds): 2700,
* number of crawler instances running at the same time: 1
* recrawl type--can be interval (recrawInterval is used) or time (recrawlTime is used): interval,
* recrawl time interval (in ISO 8601 format for durations): PT0H20M,
* recrawl time of day (in HH:mm:ss format for time): 08:00:00,
* recrawl time of day delta (in ISO 8601 format for durations); recrawl will occur at recrawlTime/recrawInterval +/- rand(recrawlTimeDelta), depending on recrawlType: PT1H.  
Those parameters were used so not to stress the news web servers with concurrent or many consecutive requests, and so not to get the crawler's/scraper's IP address blacklisted. Using the aforementioned configuration, the time between crawls is a random value between 1.8 and 2.7 s. In addition, the recrawl is triggered at a somewhat random time each day.