# Hybrid Data Set Optimization in Recommender Systems Using Fuzzy T-Norms

Antonios Papaleonidas, Elias Pimenidis, Lazaros Iliadis

► To cite this version:

Antonios Papaleonidas, Elias Pimenidis, Lazaros Iliadis. Hybrid Data Set Optimization in Recommender Systems Using Fuzzy T-Norms. 15th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), May 2019, Hersonissos, Greece. pp.647-659, 10.1007/978-3-030-19823-7_54. hal-02331301

HAL Id: hal-02331301

https://inria.hal.science/hal-02331301

Submitted on 24 Oct 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Hybrid data set optimization in recommender systems using Fuzzy T-norms

Antonios Papaleonidas¹, Elias Pimenidis² and Lazaros Iliadis³

¹,³ School of Engineering, Department of Civil Engineering, Faculty of Mathematics Programing and General courses, Democritus University of Thrace, Kimmeria, Xanthi, Greece papaleon@sch.gr¹, liliadis@civil.duth.gr³

² Department of Computer Science and Creative Technologies, University of the West of England, BS16 1QY, Bristol, United Kingdom Elias.Pimenidis@uwe.ac.uk

**Abstract.** A recommender system uses specific algorithms and techniques in order to suggest specific services, goods or other type of recommendations that users could be interested in. User's preferences or ratings are used as inputs and top-N recommendations are produced by the system. The evaluation of the recommendations is usually based on accuracy metrics such as the Mean Absolute Error (MAE) and the Root Mean Squared Error (RMSE), while on the other hand Precision and Recall is used to measure the quality of the top-N recommendations. Recommender systems development has been mainly focused in the development of new recommendation algorithms. However, one of the major problems in modern offline recommendation system is the sparsity of the datasets and the selection of the suitable users Y that could produce the best recommendations for users X. In this paper, we propose an algorithm that uses Fuzzy sets and Fuzzy norms in order to evaluate the correlation between users in the data set so the system can select and use only the most relevant users. At the same time, we are extending our previous work about Reproduction of experiments in recommender systems by developing new explanations and variables for the proposed new algorithm. Our proposed approach has been experimentally evaluated using a real dataset and the results show that it is really efficient and it can increase both accuracy and quality of recommendations.

**Keywords:** Recommender systems, Evaluation, Explanations, Reproducibility, Fuzzy logic, T-Norms

## 1 Introduction

The use of recommender systems is very common, especially in applications like e-Commerce and social networks. Products recommendation techniques can reduce the overall searching time of an e-shop user and increase sales. Apart from e-commerce, recommendation technology is also used in various other less known domains such as music recommendation [1, 4], books [2], documents [3], television programs [5], people to people recommendation [1], applications in markets [6], e-learning [7] and Web search [8].
The increasing importance, use and popularity of recommender systems research both in academia and in industry has led to the development of new algorithms and their experimental evaluation. Researchers are mostly focusing in creating more effective algorithms and models by trying to minimize the MAE and RMSE while at the same time they are trying to improve precision and recall of top-N recommendations [9, 10]. While this is important to do, it should be also noted that the problem of reproducing the results exists and it is considered important [11]. In a previous work of the research team [13], offline recommender system results were successfully reproduced using an explanation-based approach.

There are different libraries that can be used for developing and testing a recommen-
dation algorithm and include, among others, Recommender101, Apache Mahout,
LensKit and MyMediaLite [13]. Polatidis et al [1], has shown that reproducing the ex-
perimental results of an algorithm is very difficult when using a different library be-
cause of different settings and parameters that exist between them.

In this paper we are further expanding our previous work by extending Recommender101 libraries, modifying the methodology that Recommender101 is selecting users with common ratings and adding all the necessary parameters in the configuration files of Recommender101.

The rest of the paper is organized as follows: Section 2 provides the relevant back-
ground and describes related work, section 3 delivers the proposed extension and algo-
rithm of the team’s previous method, section 4 describes the experiments, the results
and the discussion while conclusions and future work are included in section 5.

## 2 Related work

Recommenders are often evaluated and compared offline using datasets collected from online platforms [18]. Evaluation can be done by using prediction accuracy or information retrieval metrics. However, the problem arises when in a research output of a new algorithm the source code is not made publicly available or when the exact settings for replicating the code and the experiments are missing [1][19].

Research papers that propose new recommendation algorithms will typically de-
scribe the experimental setup, the dataset used, and the framework that was [13]. A
major challenge in recommender system evaluation is that there are many different li-
braries for evaluating algorithms and the possibility of having one single library or
making all the current libraries following a universal or standardized approach is rather
impossible [1].

The idea of a unified approach which can facilitate a common reference baseline for recommendation experiments across different frameworks and a set of guidelines to tackle a cross-industry challenge was proposed in [1] and was the trigger point for the proposal and development of a reproducibility framework that combines the ability to reproduce recommendation experiments as well as the support of new algorithms and methodologies.
## 2.1 Reproduction of experiments in recommender systems evaluation based on explanations

In [13], the problem of reproducibility in recommender systems evaluation was highlighted and the importance of the correct settings and parameters which were used within the library was indicated.

In the proposed approach we:

1. Retrieve information from the configuration file

2. Write the information in the log file along with evaluation result and describe the written information.

The settings retrieved from the configuration file are the following and are presented in the same way that are saved in the log file:

1. The configuration parameters and settings can be set at the configuration file recommender101.properties that be found under the conf directory of Recommender101.

2. The filename of the dataset is (name of the file goes here).

3. The minimum number of ratings per user to be considered is (number).

4. The minimum number of ratings per considered item is (number).

5. This experiment has used all users OR This experiment has used (number) users.

6. The minimum rating value applied is (number e.g. between 1 to 5).

7. The maximum rating value applied is (number e.g. between 1 to 5).

8. This experiment is based on a (number e.g. 5 or 10) cross fold validation OR this experiment is based on a training/test approach using (number %) for training and (number %) for testing.

9. The number of nearest neighbors used is (number).

10. The algorithm used is (name).

11. The metrics used for this experiment are (This is already implemented in recommender101).

12. The results are (This is already implemented in recommender101).

The initial evaluation results were promising and triggered the idea of extending the configuration file and the necessary libraries of recommender101 in order to achieve a better way of selecting the most proper users Y that could produce the best recommendations for users X.

## 2.2 Reproducibility of recommender systems experiments based on explanations

On [13] extension we modified the source code of Recommender101.java, Recommender101Impl.java and DefaultDataLoader.java in order to:

* Enrich the recommender101 configuration file (recommender101.properties) with new parameters that will help us in optimizing the use of data set.
* Improve both accuracy and quality of recommendations by providing tools to select the most proper users Y that could produce the best recommendations for users X

* Strengthen our previous work by checking the offline reproduction of recommender system experiment with the use of modified source code and extended configuration file.

Five new parameters have been incorporated in the configuration file of Recommender101 and also were defined in the source code of the platform. Those parameters are:

1. **My_User_count**: Takes a value from zero, up to the total number of users found in the dataset. It works in co-operation with the already existing parameter sampleNUsers and it should be grater than sampleNUsers. It indicates the total number of users which will be used in order to select the best matches for sampleNUsers. If the value is <=0 then the modified algorithm is skipped and Recommender101 runs as usual.

2. **My:** This is a similarity value between two users. The higher the penalty multiplier the lower similarity between users with no identical ratings. For example, assuming that two users i and z have 5 common ratings with the values of the i user {3,4,2,5,1} and for z {3,2,2,5,2} with a penalty multiplier of 1 then the value returned is (5-1\*|3-3|)+(5-1\*|4-2|)+(5-1\*|2-2|)+(5-1\*|5-5|)+(5-1\*|1-2|)=22 and in the case where the penalty multiplier is 2 then the value returned is (5-2\*|3-3|)+(5-2\*|4-2|)+(5-2\*|2-2|)+(5-2\*|5-5|)+(5-2\*|1-2|)=19, which means less similar users.

3. **My_Relat_Function**: Since in a dataset many users might have submitted hundreds of ratings and other users very few there might be huge differences when the penalty multiplier is applied. Therefore, to overcome this issue, AVG or SUM can be used as parameters in My_Relat_Function. AVG will normalize the differences and SUM will keep the raw calculated values.

4. **My_fuzzy_norm**: This parameter represents the equation that calculates the overall similarity value of a user i with all other users. Sigma Count Average [14] was used in the basic implementation and in this paper Hamacher and Einstein product are introduced and used.

5. **My_fuzzy_mv**: After the fuzzy norm is calculated then this is the threshold that is used as a decision-making point for which users are similar and which are not.

By default, Recommender101 system uses a partial set of the available users in the data set in order to calculate top-N recommendations. Already existing sampleNUsers parameter is used to define the size of that partial set of users that will be used. The proposed algorithm uses the similarity between My_User_count number of users in order to select the best ones that will be included in the data set.
# 3 Proposed extension and Algorithm

DefaultDataLoader is the class of Recommender101 that is used to read the dataset that is described in the configuration file of the platform. We decided not to modify the recommendation algorithm but to work in DefaultDataLoader class. By doing that the main recommendation algorithm works on the same way and we can apply any type of source dataset manipulation without the need of rechecking the recommendation methodology. The core class of the platform is getting the new, processed, dataset and the main algorithm and the evaluation metrics remain the same.

The steps of the algorithm are the following:

1. Read from the configuration file the My_User_count, thus read the set of users U={U₁, U₂, U₃, ..., Uₓ}, the set of items UMᵢ = {(Uᵢ,M₁), (Uᵢ,M₂), ..., (Uᵢ,Mᵧ)} and the ratings for each item URᵢ= {(M₁,R), (M₂,R), ..., (Mᵧ,R)}.

2. Find all common items for each user Uₓ with all other users of the dataset U. Then, for each user i create a set X with common items, thus creating X² sets as shown in equation 1.

$$ \forall (U_i, U_z \in U), C_{i,z} = UM_i \cap UM_z \quad (1) $$

3. Apply equation 2 to calculate the matching-degree between two users i and z. In this equation y is the number of common ratings (length of Cᵢ,ₖ), UᵢMₖR is the rating of user I for an item k and UzMₖR is the rating of user z for the same item k.

$$ \forall C_{i,z}, MD_{i,z} = \sum_{k=1}^{y} (MaxRating - PM * |U_iM_kR - U_zM_kR|). \quad (2) $$

4. Step 3 returns an X * X table with matching-degree values and this step will return and Min value, a Max value, and Average (avg) value and a standard deviation (SD) value.

5. Calculate variables F and G which will be used for the fuzzification process of the matching-degree for every user combination Uᵢ, z. The calculation of F is shown in equation 3 and of G in equation 4.

$$ F = \begin{cases} Avg - (2 * SD), & \text{if } (Avg - 2(SD)) \geq \text{Min} \\ Min, & \text{if } (Avg - 2(SD)) < \text{Min} \end{cases} \quad (3) $$

$$ G = \begin{cases} Avg + (2 * SD), & \text{if } (Avg - 2(SD)) \geq \text{Max} \\ Max, & \text{if } (Avg - 2(SD)) < \text{Max} \end{cases} \quad (4) $$

6. The fuzzification will consider all Cᵢ,ₖ values based on two fuzzy sets Low and High using two symmetric semi-trapezoid function as shown in equations 5 and 6.
$$Low C_{i,z}(C_{i,z}; F, G) = \begin{cases} 1, & C_{i,z} \le F \\ \frac{G-C_{i,z}}{G-F}, & F < C_{i,z} < G \\ 0, & C_{i,z} \ge G \end{cases} \quad (5)$$

$$High C_{i,z}(C_{i,z}; F, G) = \begin{cases} 0, & C_{i,z} \le F \\ \frac{C_{i,z} - F}{G - F}, & F < C_{i,z} < G \\ 1, & C_{i,z} \ge G \end{cases} \quad (6)$$

7. The final user matching degree is calculated based on the high fuzzy set through Sigma Count average, hybrid Hamacher or Einstein products.

8. For each user the final value is checked and if it is smaller than the My_fuzzy_mv value do not pass this user to the recommendation algorithm.

9. Start Recommender101 based on the settings and algorithms found in the configuration file.

## 3.1 Hybrid Hamacher and Einstein products

Hamacher and Einstein products are two well known Fuzzy T-Norms [20] that are widely used in Fuzzy Information Aggregation [15][16] and they are calculated based on the equations 7 (Hamacher product) and 8 (Einstein product).

$$\bar{A} \cap \bar{B} = \frac{\mu_{\bar{A}}(x)\mu_{\bar{B}}(x)}{\mu_{\bar{A}}(x) + \mu_{\bar{B}}(x) - \mu_{\bar{A}}(x)\mu_{\bar{B}}(x)} \quad (7)$$

$$\bar{A} \cap \bar{B} = \frac{\mu_{\bar{A}}(x)\mu_{\bar{B}}(x)}{2 - [\mu_{\bar{A}}(x) + \mu_{\bar{B}}(x) - \mu_{\bar{A}}(x)\mu_{\bar{B}}(x)]} \quad (8)$$

In both cases membership values equal to Zero will result both products to be equal to zero. In that case relativity of the user will be also zero so in our hybrid version of Hamacher and Einstein products, zero membership values are not used in the calculations. This modification can be adopted due to the existence of custom variable My_Relat_Function which will “promote” users with high number of ratings even if they got some zeros in membership values.

After calculating the desired product for each user, the values are linearly normalized in [0,1] in order to be in match with My_fuzzy_mv threshold.

# 4 Experiments and results

The proposed methodology was extensively tested by measuring the performance and the accuracy of the Recommender101 under different configuration schemas.

At first part of testing we used the system with the default configuration and without processing the dataset. Four different values of sampleNUsers (100, 400, 800 and 1600) were used.
At the second part of testing we used the system by enabling the source dataset pro-
cessing with different values for My_User.count and My_fuzzy.norm custom variables.
Once again, each case was tested with the same four different values of sampleNUsers
as before. The value of My_User.count valiable was set to 300, 1200, 2400, 4600 users
accordingly to sampleNUsers. Penalty multiplier was set to 1.7, Fuzzy_mv threshold
was set to 0.5 and realativity function was set to AVG. All three alternatives for
My_fuzzy.norm, Sigma Count average, hybrid Hamacher or Einstein products, were
also used.

In total 16 different configurations were used, and the same number of results were
created. In all cases both “FunkSVDRecommender” and “SlopeOneRecommender” al-
gorithms were used.

Custom variable's values were set based on the fact that this paper focuses in the
importance of source dataset prospecting and not in finding the best values for achiev-
ing the maximum accuracy.

MovieLens 1 million dataset [17], which consists of 6040 users, 4000 movies and 1
million ratings in a 1-5 scale was used for the experimental evaluation of the method.

All experiment took place on a Windows 10 64bit computer with 64GB ram and an
i5-4570 cpu.

## 4.1 Evaluation metrics

The evaluation was based on well-known and widely used metrics like Mean Absolute
Error (MAE) and the Root Mean Squared Error (RMSE) for error rating, and Precision
and Recall for measuring the quality of the top-N recommendations [1,11,13].

MAE and RMSE are rating error prediction metrics [20] and lower values represent better predictions, whereas Precision and Recall are information retrieval metrics and represent the quality of the retrieved recommendations and higher values are better.

MAE, RMSE, precision and recall equations are following.

$$MAE = \frac{1}{n} \sum_{i=1}^{n} |pi - ri| \quad (9)$$

$$RMSE = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (p_i - r_i)} \quad (10)$$

$$precision = \frac{\text{Correctly recommended items}}{\text{Total recommender items}} \quad (11)$$

$$recall = \frac{\text{Correctly recommended items}}{\text{Relevant items}} \quad (12)$$

Execution Duration is measured in seconds and was used for performance evaluation
of the proposed method.
## 4.2 Experimental results

First four sets of results, table 1, refer to first run of the system with the default Recommender101 configuration, with no dataset process and values of sampleNUsers equal to 100, 400, 800 and 1600.

<table><caption>Table 1. Default Recommender101 execution.</caption><thead><tr><th>sampleNUsers</th><th>100</th><th>400</th><th>800</th><th>1600</th></tr></thead><tbody><tr><td>Items</td><td>3706</td><td>3706</td><td>3706</td><td>3706</td></tr><tr><td>Ratings</td><td>14720</td><td>60796</td><td>137742</td><td>274529</td></tr><tr><td>Sparsity</td><td>0,04</td><td>0,041</td><td>0,046</td><td>0,046</td></tr><tr><td>Avg. Ratings/User</td><td>147,2</td><td>151,99</td><td>172,178</td><td>171,581</td></tr><tr><td>Avg. Ratings/Item</td><td>3,972</td><td>16,405</td><td>37,167</td><td>74,077</td></tr><tr><td>Max /User</td><td>1050</td><td>1277</td><td>1302</td><td>1743</td></tr><tr><td>Max /item</td><td>59</td><td>219</td><td>469</td><td>917</td></tr><tr><td>MAE FunkSVD</td><td>0,802</td><td>0,739</td><td>0,725</td><td>0,72</td></tr><tr><td>MAE SlopeOne</td><td>0,792</td><td>0,733</td><td>0,711</td><td>0,692</td></tr><tr><td>RMSE FunkSVD</td><td>1,024</td><td>0,94</td><td>0,921</td><td>0,915</td></tr><tr><td>RMSE SlopeOne</td><td>1,004</td><td>0,933</td><td>0,904</td><td>0,882</td></tr><tr><td>Precision FunkSVD</td><td>0,693</td><td>0,724</td><td>0,742</td><td>0,755</td></tr><tr><td>Precision SlopeOne</td><td>0,691</td><td>0,724</td><td>0,735</td><td>0,739</td></tr><tr><td>Recall FunkSVD</td><td>0,488</td><td>0,514</td><td>0,518</td><td>0,523</td></tr><tr><td>Recall SlopeOne</td><td>0,483</td><td>0,517</td><td>0,509</td><td>0,509</td></tr><tr><td>Execution Duration</td><td>22</td><td>133</td><td>294</td><td>632</td></tr></tbody></table>

Next results, table 2, are showing the use of the proposed method with Sigma Count average as the final fuzzification method.

<table><caption>Table 2. Sigma Count average as final fuzzification method results.</caption><thead><tr><th>sampleNUsers / My_User.count</th><th>100 (300)</th><th>400 (1200)</th><th>800 (2400)</th><th>1600 (4800)</th></tr></thead><tbody><tr><td>Items</td><td>2972</td><td>3418</td><td>3576</td><td>3662</td></tr><tr><td>Ratings</td><td>13137</td><td>67405</td><td>134833</td><td>276467</td></tr><tr><td>Sparsity</td><td>0,044</td><td>0,049</td><td>0,047</td><td>0,047</td></tr><tr><td>Avg. Ratings/User</td><td>131,37</td><td>168,512</td><td>168,541</td><td>172,792</td></tr><tr><td>Avg. Ratings/Item</td><td>4,42</td><td>19,721</td><td>37,705</td><td>75,496</td></tr><tr><td>Max /User</td><td>608</td><td>1518</td><td>1323</td><td>1271</td></tr><tr><td>Max /item</td><td>61</td><td>255</td><td>514</td><td>961</td></tr><tr><td>MAE FunkSVD</td><td>0,773</td><td>0,717</td><td>0,705</td><td>0,707</td></tr><tr><td>MAE SlopeOne</td><td>0,765</td><td>0,713</td><td>0,692</td><td>0,682</td></tr><tr><td>RMSE FunkSVD</td><td>0,991</td><td>0,912</td><td>0,894</td><td>0,898</td></tr><tr><td>RMSE SlopeOne</td><td>0,972</td><td>0,917</td><td>0,879</td><td>0,868</td></tr><tr><td>Precision FunkSVD</td><td>0,714</td><td>0,737</td><td>0,751</td><td>0,763</td></tr><tr><td>Precision SlopeOne</td><td>0,71</td><td>0,736</td><td>0,744</td><td>0,744</td></tr></tbody></table>
<table><tr><td>Recall FunkSVD</td><td>0,519</td><td>0,517</td><td>0,527</td><td>0,524</td></tr><tr><td>Recall SlopeOne</td><td>0,504</td><td>0,519</td><td>0,516</td><td>0,512</td></tr><tr><td>Execution Duration</td><td>15</td><td>136</td><td>265</td><td>385</td></tr></table>

Hybrid Hamacher product results are in table 3 below.

<table><caption>Table 3. Hybrid Hamacher product as final fuzzification method results.</caption><thead><tr><th>sampleNUsers /<br>My_User.count</th><th>100<br>(300)</th><th>400<br>(1200)</th><th>800<br>(2400)</th><th>1600<br>(4800)</th></tr></thead><tbody><tr><td>Items</td><td>2931</td><td>3375</td><td>3507</td><td>3642</td></tr><tr><td>Ratings</td><td>12997</td><td>67302</td><td>135107</td><td>278365</td></tr><tr><td>Sparsity</td><td>0,044</td><td>0,049</td><td>0,048</td><td>0,048</td></tr><tr><td>Avg. Ratings/User</td><td>131,37</td><td>168,512</td><td>168,541</td><td>172,792</td></tr><tr><td>Avg. Ratings/Item</td><td>4,46</td><td>19,714</td><td>37,788</td><td>79,14</td></tr><tr><td>Max /User</td><td>685</td><td>1571</td><td>1334</td><td>1420</td></tr><tr><td>Max /item</td><td>68</td><td>276</td><td>603</td><td>1007</td></tr><tr><td>MAE FunkSVD</td><td>0,761</td><td>0,711</td><td>0,699</td><td>0,703</td></tr><tr><td>MAE SlopeOne</td><td>0,759</td><td>0,711</td><td>0,689</td><td>0,679</td></tr><tr><td>RMSE FunkSVD</td><td>0,981</td><td>0,91</td><td>0,899</td><td>0,897</td></tr><tr><td>RMSE SlopeOne</td><td>0,972</td><td>0,918</td><td>0,861</td><td>0,86</td></tr><tr><td>Precision FunkSVD</td><td>0,723</td><td>0,74</td><td>0,767</td><td>0,766</td></tr><tr><td>Precision SlopeOne</td><td>0,723</td><td>0,736</td><td>0,741</td><td>0,748</td></tr><tr><td>Recall FunkSVD</td><td>0,52</td><td>0,518</td><td>0,531</td><td>0,528</td></tr><tr><td>Recall SlopeOne</td><td>0,512</td><td>0,519</td><td>0,525</td><td>0,519</td></tr><tr><td>Execution Duration</td><td>16</td><td>142</td><td>301</td><td>410</td></tr></tbody></table>

Hybrid Einstein product results are in table 4 below.

<table><caption>Table 4. Hybrid Einstein product as final fuzzification method results.</caption><thead><tr><th>sampleNUsers /<br>My_User.count</th><th>100<br>(300)</th><th>400<br>(1200)</th><th>800<br>(2400)</th><th>1600<br>(4800)</th></tr></thead><tbody><tr><td>Items</td><td>2972</td><td>3418</td><td>3576</td><td>3662</td></tr><tr><td>Ratings</td><td>13137</td><td>67405</td><td>134833</td><td>276467</td></tr><tr><td>Sparsity</td><td>0,044</td><td>0,049</td><td>0,047</td><td>0,047</td></tr><tr><td>Avg. Ratings/User</td><td>131,37</td><td>168,512</td><td>168,541</td><td>172,792</td></tr><tr><td>Avg. Ratings/Item</td><td>4,42</td><td>19,721</td><td>37,705</td><td>75,496</td></tr><tr><td>Max /User</td><td>680</td><td>1603</td><td>1371</td><td>1407</td></tr><tr><td>Max /item</td><td>72</td><td>268</td><td>530</td><td>980</td></tr><tr><td>MAE FunkSVD</td><td>0,76</td><td>0,714</td><td>0,701</td><td>0,69</td></tr><tr><td>MAE SlopeOne</td><td>0,756</td><td>0,709</td><td>0,671</td><td>0,677</td></tr><tr><td>RMSE FunkSVD</td><td>0,983</td><td>0,904</td><td>0,89</td><td>0,891</td></tr><tr><td>RMSE SlopeOne</td><td>0,972</td><td>0,917</td><td>0,879</td><td>0,868</td></tr><tr><td>Precision FunkSVD</td><td>0,722</td><td>0,739</td><td>0,761</td><td>0,764</td></tr><tr><td>Precision SlopeOne</td><td>0,717</td><td>0,738</td><td>0,748</td><td>0,751</td></tr></tbody></table>
<table><tr><td>Recall FunkSVD</td><td>0,523</td><td>0,519</td><td>0,529</td><td>0,531</td></tr><tr><td>Recall SlopeOne</td><td>0,511</td><td>0,52</td><td>0,531</td><td>0,527</td></tr><tr><td>Execution Duration</td><td>16</td><td>142</td><td>298</td><td>409</td></tr></table>

Next three tables show the percentage difference on basic evaluation metrics between standard Recommender101 execution and Sigma Count, Hybrid Hamacher product and Hybrid Einstein product.

<table><caption>Table 5. Hybrid Einstein product as final fuzzification method results.</caption><thead><tr><th>sampleNUsers /<br>My_User.count</th><th>100<br>(300)</th><th>400<br>(1200)</th><th>800<br>(2400)</th><th>1600<br>(4800)</th></tr></thead><tbody><tr><td>MAE FunkSVD</td><td>-3,62%</td><td>-2,98%</td><td>-2,76%</td><td>-1,81%</td></tr><tr><td>MAE SlopeOne</td><td>-3,41%</td><td>-2,73%</td><td>-2,67%</td><td>-1,45%</td></tr><tr><td>RMSE FunkSVD</td><td>-3,22%</td><td>-2,98%</td><td>-2,93%</td><td>-1,86%</td></tr><tr><td>RMSE SlopeOne</td><td>-3,19%</td><td>-1,71%</td><td>-2,77%</td><td>-1,59%</td></tr><tr><td>Precision FunkSVD</td><td>3,03%</td><td>1,80%</td><td>1,21%</td><td>1,06%</td></tr><tr><td>Precision SlopeOne</td><td>2,75%</td><td>1,66%</td><td>1,22%</td><td>0,68%</td></tr><tr><td>Recall FunkSVD</td><td>6,35%</td><td>0,58%</td><td>1,74%</td><td>0,08%</td></tr><tr><td>Recall SlopeOne</td><td>4,35%</td><td>0,39%</td><td>1,38%</td><td>0,59%</td></tr></tbody></table>

<table><caption>Table 6. Hybrid Einstein product as final fuzzification method results.</caption><thead><tr><th>sampleNUsers /<br>My_User.count</th><th>100<br>(300)</th><th>400<br>(1200)</th><th>800<br>(2400)</th><th>1600<br>(4800)</th></tr></thead><tbody><tr><td>MAE FunkSVD</td><td>-5,11%</td><td>-3,79%</td><td>-3,59%</td><td>-2,36%</td></tr><tr><td>MAE SlopeOne</td><td>-4,17%</td><td>-3,00%</td><td>-3,09%</td><td>-1,88%</td></tr><tr><td>RMSE FunkSVD</td><td>-4,20%</td><td>-3,19%</td><td>-2,39%</td><td>-1,97%</td></tr><tr><td>RMSE SlopeOne</td><td>-3,19%</td><td>-1,61%</td><td>-4,76%</td><td>-2,49%</td></tr><tr><td>Precision FunkSVD</td><td>4,33%</td><td>2,21%</td><td>3,37%</td><td>1,46%</td></tr><tr><td>Precision SlopeOne</td><td>4,63%</td><td>1,66%</td><td>0,82%</td><td>1,22%</td></tr><tr><td>Recall FunkSVD</td><td>6,56%</td><td>0,78%</td><td>2,51%</td><td>0,96%</td></tr><tr><td>Recall SlopeOne</td><td>6,00%</td><td>0,39%</td><td>3,14%</td><td>1,96%</td></tr></tbody></table>

<table><caption>Table 7. Hybrid Einstein product as final fuzzification method results.</caption><thead><tr><th>sampleNUsers /<br>My_User.count</th><th>100<br>(300)</th><th>400<br>(1200)</th><th>800<br>(2400)</th><th>1600<br>(4800)</th></tr></thead><tbody><tr><td>MAE FunkSVD</td><td>-5,24%</td><td>-3,38%</td><td>-3,31%</td><td>-4,17%</td></tr><tr><td>MAE SlopeOne</td><td>-4,55%</td><td>-3,27%</td><td>-5,63%</td><td>-2,17%</td></tr><tr><td>RMSE FunkSVD</td><td>-4,00%</td><td>-3,83%</td><td>-3,37%</td><td>-2,62%</td></tr><tr><td>RMSE SlopeOne</td><td>-3,19%</td><td>-1,71%</td><td>-2,77%</td><td>-1,59%</td></tr><tr><td>Precision FunkSVD</td><td>4,18%</td><td>2,07%</td><td>2,56%</td><td>1,19%</td></tr><tr><td>Precision SlopeOne</td><td>3,76%</td><td>1,93%</td><td>1,77%</td><td>1,62%</td></tr><tr><td>Recall FunkSVD</td><td>7,17%</td><td>0,97%</td><td>2,12%</td><td>1,53%</td></tr><tr><td>Recall SlopeOne</td><td>5,80%</td><td>0,58%</td><td>4,32%</td><td>3,54%</td></tr></tbody></table>
## 4.3 Discussion

We can see from the tables above that in all the cases that the extended DefaultData-Loader class of Recommender101 was used the recommendation results have been improved. Data sparsity has improved up to 20% (from 0.041 to 0.049) in the experiments where 400 sampleNUsers were provided to Recommender101. MAE has improved by 1.45% to 5.63% and RMSE improved by 1.59% to 4.76%. While checking the quality of recommendations we can see that Precision has improved by 0.68% to 4.63% and Recall by 0.08% to 7.17%.

At the same time, we can see that the execution duration of the recommendation algorithm drops up to 40% (from 632sec to 385 sec) while only in few cases it can rise only up to 6% (from 133 sec to 136 sec).

Although that, as mentioned before, the selection of the optimal values for the new custom valuables is not the main scope of this paper we run an extra experiment by using the Hybrid Einstein product as final fuzzification method and setting the fuzzy membership value (FMV) threshold to 0.55 instead of 0.50. We run the experiment only for the combination of 1600/4800 because in that case we saw that improvement was mainly reduced compared to the other three combinations. The following table 8 shows the results of that extra experiment compared with the results with the default run of Recommender101.

<table><caption>**Table 8.** Comparison of experimental results of run with FMV threshold equal to 0.55 and standard Recommender101 run.</caption><thead><tr><th>sampleNUsers / My_User.count</th><th>Extra experi-ment</th><th>Default run</th><th>Difference</th></tr></thead><tbody><tr><td>Items</td><td>3574</td><td>3706</td><td>-132</td></tr><tr><td>Ratings</td><td>337632</td><td>274529</td><td>63103</td></tr><tr><td>Sparsity</td><td>0,059</td><td>0,046</td><td>28,26%</td></tr><tr><td>Avg. Ratings/User</td><td>211,02</td><td>171,581</td><td>39,439</td></tr><tr><td>Avg. Ratings/Item</td><td>94,469</td><td>74,077</td><td>20,392</td></tr><tr><td>Max /User</td><td>1743</td><td>1743</td><td>0</td></tr><tr><td>Max /item</td><td>1100</td><td>917</td><td>183</td></tr><tr><td>MAE FunkSVD</td><td>0,651</td><td>0,72</td><td>-9,58%</td></tr><tr><td>MAE SlopeOne</td><td>0,629</td><td>0,692</td><td>-9,10%</td></tr><tr><td>RMSE FunkSVD</td><td>0,826</td><td>0,915</td><td>-9,73%</td></tr><tr><td>RMSE SlopeOne</td><td>0,8</td><td>0,882</td><td>-9,30%</td></tr><tr><td>Precision FunkSVD</td><td>0,787</td><td>0,755</td><td>4,24%</td></tr><tr><td>Precision SlopeOne</td><td>0,773</td><td>0,739</td><td>4,60%</td></tr><tr><td>Recall FunkSVD</td><td>0,531</td><td>0,523</td><td>1,53%</td></tr><tr><td>Recall SlopeOne</td><td>0,524</td><td>0,509</td><td>2,95%</td></tr><tr><td>Execution Duration</td><td>713</td><td>632</td><td>12,82%</td></tr></tbody></table>
We can see from table above that by increasing the FMV threshold, all the evaluation metrics are improved, and the sparsity of the dataset is dramatically increased. Of course, in case of a very high FMV threshold system could face the problem of not being able to select the minimum number of users required to run the recommendation algorithm.

# 5 Conclusions and future work

In this paper an extended methodology of reproducibility in recommender systems combined with source dataset optimization methods with the use or Recommender101 platform has been presented. In our previous work we have already shown that the reproducibility of results becomes achievable if the correct settings and parameters are used. This paper extends the set of parameters and achieves to increase the recommendation accuracy, to deal with the sparsity problem of big datasets and at the same time to provide the necessary tools and variables to reproduce the results of the experiments.

The initial evaluation results are promising and can assist towards this direction and our approach can be straightforwardly implemented by researchers in other libraries. Furthermore, in our future work we aim to include more custom variables in the Recommender101 and to extend the use of Fuzzy Norms in the user selection methodology.

# References

1. Polatidis, N., Kapetanakis, S., Pimenidis, E., & Kosmidis, K. Reproducibility of experiments in recommender systems evaluation 14th International Conference on Artificial Intelligence Applications and Innovations, AIAI 2018. IFIP AICT 519 pp.

2. Núñez-Valdéz, E.R., et al., Implicit feedback techniques on recommender systems applied to electronic books. Computers in Human Behavior, 2012. 28(4): p. 1186-1193.

3. Porcel, C., et al., A hybrid recommender system for the selective dissemination of research resources in a technology transfer office. Information Sciences, 2012. 184(1): p. 1-19.

4. Tan, S., et al., Using rich social media information for music recommendation via hyper-graph model. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2011. 7(1): p. 22.

5. Barragáns-Martínez, A.B., et al., A hybrid content-based and item-based collaborative filtering approach to recommend TV programs enhanced with singular value decomposition. Information Sciences, 2010. 180(22): p. 4290-4311.

6. Costa-Montenegro, E., A.B. Barragáns-Martínez, and M. Rey-López, Which App? A recommender system of applications in markets: Implementation of the service for monitoring users' interaction. Expert systems with applications, 2012. 39(10): p. 9367-9375.

7. Bobadilla, J., F. Serradilla, and A. Hernando, Collaborative filtering adapted to recommender systems of e-learning. Knowledge-Based Systems, 2009. 22(4): p. 261-265.

8. McNally, K., et al., A case study of collaboration and reputation in social web search. ACM Transactions on Intelligent Systems and Technology (TIST), 2011. 3(1): p. 4.

9. Jalili, M., et al., Evaluating Collaborative Filtering Recommender Algorithms: A Survey. IEEE Access, 2018. 6: p. 74003-74024.

10. Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22, 5-53 (2004).
11. Said, A., Bellogín, A.: Comparative recommender system evaluation. Proc. 8th ACM Conf. Recomm. Syst. - RecSys '14. 129–136 (2014).

12. Jannach, D., Lerche, L., Gedikli, F., Bonnin, G.: What recommenders recommend–an analysis of accuracy, popularity, and sales diversity effects. In: User Modeling, Adaptation, and Personalization. pp. 1–13 (2013).

13. Polatidis Nikolaos, Pimenidis Elias. Reproduction of experiments in recommender systems evaluation based on explanations. In: International Conference on Engineering Applications of Neural Networks. Springer, Cham, 2018. p. 194-200.

14. Zadeh, L. A. (1988). Fuzzy logic. Computer, 21(4), 83-93.

15. Wang, W., & Liu, X. (2011). Intuitionistic fuzzy geometric aggregation operators based on Einstein operations. International Journal of Intelligent Systems, 26(11), 1049-1075.

16. Liu, P. (2014). Some Hamacher aggregation operators based on the interval-valued intuitionistic fuzzy numbers and their application to group decision making. IEEE Transactions on Fuzzy systems, 22(1), 83-97.

17. Harper, F.M., Konstan, J.A.: The MovieLens Datasets. ACM Trans. Interact. Intell. Syst. 5, 1–19 (2015).

18. YANG, Longqi, et al. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In: Proceedings of the 12th ACM Conference on Recommender Systems. ACM, 2018. p. 279-287.

19. Polatidis, N., & Pimenidis, E. (2018, September). Reproduction of experiments in recommender systems evaluation based on explanations. In International Conference on Engineering Applications of Neural Networks (pp. 194-200). Springer, Cham.

20. Iliadis L, Papaleonidas A. Computational intelligence and intelligent agents. Thessaloniki, Greece. 2016:448.