## 4 Scalable Application Using ParSoDA  
ParSoDA (Parallel Social Data Analytics) [10] is a library for processing and analyzing in parallel large amounts of data collected from social media platforms. The main goal of ParSoDA is to allow programmers to easily extract knowledge from social media data by hiding the difficulty of defining a parallel/distributed application made up of many steps. In fact, programmers have a number of hurdles when designing and implementing these applications, including parallelizing complex algorithms, lowering communication costs, and optimizing memory utilization. For this purpose, the library allows to define a data analysis application starting from a general structure consisting of seven steps:  
1. _Data acquisition_ for collecting social media items and storing them in a persistent repository (e.g., HDFS [22]).
2. _Data filtering_ for filtering social media items according to a set of functions.
3. _Data mapping_ for transforming the information contained in each social media item through some functions.  
4. _Data partitioning_ for partitioning items into shards by a primary key and then sorting them by a secondary key.
5. _Data reduction_ for aggregating items contained in a shard according to a function.
6. _Data analysis_ for analyzing data using a data analysis function to extract the knowledge of interest.
7. _Data visualization_ for visualizing data analysis results in a suitable visual format.  
ParSoDA provides a predefined set of functions for each step. For example, ParSoDA includes functions for crawling data from Twitter and Flickr, for filtering posts based on location or time of publication, for transforming data from one format to another, for classifying and clustering data, and so on. However, users can extend this set of functions with their own.  
Listing 3 shows the code of the application for extracting the user polarization between rival political factors. First, at line 3, a _SocialDataApp_ is instantiated. Then, the output path, the file system, and a file containing the keywords used to support the two factions are defined (_lines 4-7_). The rest of the code declares the functions to be used for data filtering, data mapping, data sorting, data reduction, and data analysis.  
```
1 publicclassUserPolarizationMain{
2 publicstaticvoidmain(String[]args){
3 SocialDataAppAppapp=newSocialDataApp(\"2 FactionUserPolarization\");
4 app.setOutputBasePath(\"outputApp\");
5 app.setLocatFileSystem();
6 String[]CFiles=\"resources/twoFactionKeywords.json\"];
7 app.setDistributedCacheFiles(cFiles);
8 Class[]CFunctions=[FileReaderCrawler.class];
9 String[]cParams={\"-iresources/tweetsFinal.json\"};
10 app.setCrawlers(cFunctions,cParams);
11 Class[]mFunctions=[ClassifyTwoFactionsEvent.class];
12 String[]mParams={\"-fovectionKeywords.json\"};
13 app.setCrawlerFunctions=Functions/mParams);
14 StringgroupKey=\"userId\";
15 StringsortKey=\"DATETIME\";
16 app.setPartitioningKeys(groupKey,sortKey);
17 ClassrFunction=ReduceByTwoFactionsPolarization.class;
18 StringrParams=\"-t5\";
19 app.setReduceFunction(rFunction,rParams);
20 ClassaFunction=TworFactionsPolarization.class;
21 StringaParams=null;
22 app.setAnalysisFunction(aFunction,aParams);
23 app.execute();
24}
25}
```  
**Listing 3**. A user polarization application written using ParSoDA.  
Using ParSoDA reduces the number of lines of code required to develop complex data analysis applications, as proved in [23]. In particular, ParSoDA allows programmers to save hundreds of lines of code in the main (as they only need to configure the functions to be used at each step) and in the other steps, where built-in functionalities are used and where the programmer needs only to define the function logic. Starting from the application main, ParSoDA creates and executes a chain of tasks for the different steps. Without ParSoDA, programmers must manually control the execution and parallelization of each step.