Recommending human resources to project leaders using a collaborative filtering‐based recommender system: Case study of gitHub

IET SoftwareVolume 13, Issue 5 p. 379-385 Research ArticleOpen Access Recommending human resources to project leaders using a collaborative filtering-based recommender system: Case study of gitHub Shohreh Ajoudanian, Corresponding Author shajoudanian@pco.iaun.ac.ir orcid.org/0000-0001-8798-787X Faculty of Computer Engineering, Najafabad Branch, Islamic Azad University, Najafabad, IranSearch for more papers by this authorMaryam Nooraei Abadeh, Department of Computer Engineering, Abadan Branch, Islamic Azad University, Abadan, IranSearch for more papers by this author Shohreh Ajoudanian, Corresponding Author shajoudanian@pco.iaun.ac.ir orcid.org/0000-0001-8798-787X Faculty of Computer Engineering, Najafabad Branch, Islamic Azad University, Najafabad, IranSearch for more papers by this authorMaryam Nooraei Abadeh, Department of Computer Engineering, Abadan Branch, Islamic Azad University, Abadan, IranSearch for more papers by this author First published: 01 October 2019 https://doi.org/10.1049/iet-sen.2018.5261Citations: 6AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Recommender systems (RSs) are a significant subclass of the information filtering system. RSs seek to predict the rating or preference that a user would give to an item in various online application community fields. Collaborative filtering (CF) is a technique which predicts user distinctions by learning past user-item relationships. However, it is hard to perceive the comparable interests between customers in light of the fact that the sparsity problem is caused by the deficient number of the relationship between users. It is a challenge which limited the ease of use of CF. This paper proposes a novel fuzzy C-means clustering approach which is used to deal with this sparsity problem by utilising a sparsest sub-graph detection algorithm in defining initial centres of the clustering method. The approach uses adaptability of fuzzy logic to make better personalised recommendations in terms of precision, recall and F-measure. The authors present a case study where GitHub is used to show the effectiveness of authors' approach. Authors' model can recommend relevant human resources (HR) to project leaders who have participated in similar projects. The comparative experiment results show that the planned approach will effectively solve the sparseness drawback and produce suitable coverage rate and recommendation quality. 1 Introduction With the enormous growth of Internet services such as e-commerce sites, recommendation systems became an important research area. With the increase in diversity of products and services, consumers can gain the opportunity to find ideal items along with the drawback of selecting among the profusion of uninteresting products; For example, there are several thousand films in media services provider like Netflix, a huge number of books in Amazon, in excess of ten billion page collection in social bookmarking web services such as, Del.icio.us. It is impossible to give all of the information once-over. Depending on the past transactions of a user, recommending products of interest to the user may be useful for the system and for the user. One category of recommendation systems, which solely relies on the past behaviour (ratings, purchase history, time spent) of the users, is collaborative filtering (CF). A major issue that limits the value of CF is the sparsity issue. It mentions a condition which value-based or input information is inadequate to recognise similarities in user interests [1]. The CF systems are divided into two classes: memory-based CF and model-based CF. The former plans to recognise the surroundings of users with comparative tastes and gives an indicator to the dynamic users utilising the accessible user-item data. The latter uses the user-item database to finish up a model which is then used for predictions. Furthermore, CF becomes a key process in scenarios with information overload. In order to manage such uncertainty and overload in RSs contexts, the fuzzy approach has been utilised in the related literature. Recent studies show that benefiting of fuzzy logic in personalised recommendation can supply better outcomes. Fuzzy C-means is a clustering type, which allows items to become a member in two or more clusters with various membership degrees. Here, a model-based CF approach by using a fuzzy graph clustering algorithm is proposed to deal with HR [2, 3]. The project leaders of software development organisations must give careful consideration in assigning best employee for a specific role in a software project. As a contextual analysis, this work investigates the utilisation of recommender systems RSs in suggesting HRs to project leaders to select those employees who are best suited to participating in a new project. The key contributions of this paper include: We proposed a CF method using a novel modified fuzzy C-means clustering approach, Our RS applied on GitHub data as a HR recommender to project leaders, Correlation measures i.e. Pearson correlation coefficient has been applied on GitHub data to determine and cross-check similar users and similar projects on GitHub. The paper is structured as follows. Section 2 reviews related work. An itemised description of the proposed method is presented in Section 3. In Section 4, we present a case study where GitHub is used to show the effectiveness of our approach. The evaluations of our method are presented in section 5. Finally, some concluding comments and discussion are made in Section 6. 2 Background In the current years, the study of recommendation models has become a research hotspot in various online application communities field like books, movies, and documents. There are three categories of recommendation models which are the content-based recommendation models, the CF recommendation models, and the hybrid recommendation models [4]. Content-based recommendation utilises users' historical task records to create a recommendation model. It computes similarities of users or extracts features from task information through statistics and machine learning methods. Such recommendation methods work well for structured information (news, articles etc.), but have poor performance for unstructured information. CF recommendation models recommend products to users based on similar assessment between target users and other users [5]. As we will discuss in follow sub-section, in spite of the fact that the CF method is employed in numerous applications, there are still some unsolved problems, such as sparsity and scalability. In hybrid recommendation models, the previously stated algorithms combine to create a new algorithm. A recent survey paper [6] has shown that some traditional fuzzy tools have been successfully used for a more flexible and accurate information processing in RSs. 2.1 Collaborative filtering recommendation models Cluster models, Bayesian Network models, and specialised association-rule algorithms among other techniques used in this field can aggregate the experiences of similar users in the system for creating recommendation in CF [7, 8]. Notwithstanding cluster filtering success in several application settings, the CF approach still have been reported to possess many major restrictions as well as the sparsity, scalability, and synonymy problems [9]. Several algorithms have been presented to deal with the CF problems. Two categories of CF algorithms are [10]: memory-based algorithms and model-based algorithms. The memory-based algorithms initially find the users from the training database that are most similar to the current test user in terms of the rating pattern, and then combine the ratings given by those similar users to obtain the prediction for the test user. Pearson correlation and cosine of the angle are the two most common methods. Model-based algorithm collects rating data to review, infer the user's action model, and predicate rating for a product. The distinction between model-based CF and memory-based CF is that the model-based approach is based on data application statistics and machine learning to get model to predicates. 2.2 Clustering in recommender systems Clustering techniques work by identifying groups of users who seem to have similar preferences. Once the clusters are created, by averaging the opinions of the other users in that cluster, predictions for an individual can be made. Some clustering techniques represent each user with partial participation in several clusters. The prediction is then an average across the clusters, weighted by the rank of participation [10]. Many hybrid RSs that use clustering algorithms have been proposed in the literature to defeat RSs' limitations [11-13]. In [11, 12], the K-means clustering algorithm is used to cluster users and items. In their algorithm, the users of a CF system are divided using K-means clustering algorithm and use the divide as neighbourhoods. The K-means cluster the user into groups called as cluster centres. In [13], a modified clustering based CF algorithm is presented. Authors here estimate the absent ratings in the user-item matrix, for calculating the similarity of items using the K-means algorithm. Authors use fuzzy logic to enhance personalisation techniques in recent researches [14]. In fuzzy clustering methods, objects can belong to two or more cluster with different membership degree while the K-means clustering algorithm divides a set of objects into a specified number k of clusters that are aimed at minimising the dissimilarity between data assigned to the same cluster. A novel modified fuzzy C-means clustering algorithm is proposed in [15], which is used for hybrid personalised RS. This algorithm has two phases. In the first phase, opinions from users are collected in the form of user-item rating matrix. In the second phase, recommendations are created online for active users using similarity measures. Moreover, in [16], the authors applied the fuzzy C-means clustering algorithm to cluster the users. The obtained clusters contribute to providing the recommendation, so that new items are suggested to the target user by evaluating the average ratings of the unrated items. In [17], a new fuzzy clustering approach based on genetic algorithms is introduced. A two-layer graph is created. After applying this clustering algorithm to the graph, the similarity between web pages and users is calculated, and recommendations using the content-based, collaborative and hybrid approaches are proposed. The authors in [18] proposed an explicit trust and distrust clustering-based CF recommendation. Their approach has good results in the cold start situation. Our contribution here is to develop an effective approach to making high-quality recommendations even when insufficient data available. The next section will discuss the sparsity problem in detail. 2.3 Sparsity problem In CF systems, we introduced users by the items they have purchased or rated. For instance, in a media service provider that store one million films, a Boolean vector of one million items represents each client. The value of each item is determined by whether this client has purchased the corresponding film in past transactions. For example, the value of 1 shows that we had a purchase and 0 shows that no purchase has occurred. When multiple clients are involved, a matrix that is created from all vectors representing these clients can be used to capture past transactions. We now introduce some notation to be used throughout the paper. We use C to denote the set of clients and P the set of items. We denote the client–product interaction matrix by a |C| × |P| matrix A = (aij), such that (1)In large-scale applications that both the number of items, |P|, and the number of clients, |C|, are large, several transactions have been recorded. Even in this condition, the clients-product interaction matrix can still be very sparse. That is very few items in A has the value of 1. This drawback commonly referred to as the sparsity issue has a major negative impact on the effectiveness of a CF approach. As of sparsity, it is extremely probable that the similarity (or correlation) between two given users is zero, rendering CF useless [19]. Even for pairs of users that are absolutely correlated, such correlation measures may not be reliable. In [20], to overcome the sparsity problem, authors have employed association rules mining technique to discover similar interest patterns among users from implicit information instead of the explicit information. In [21], authors generate target data from a linear regression of auxiliary feedback, and from the nearest neighbours with a set of purchased items in multiple dimensions; they propose a ranking model to accommodate both the original and generated data. Authors in [4] propose a recommendation method based on CF using ontology and dimensionality reduction techniques to improve the sparsity in CF. The hybrid method outperformed the nearest neighbour method in all datasets. In addition, corresponding results showed that the methods which use ontology have better prediction accuracy compared to the other methods for the dataset that is sparser. 3 Proposed method We propose a model-based CF approach by using a fuzzy graph clustering algorithm (See Fig. 1). The proposed method consists of five steps: In the first step, the set of users/items mapped into a graph. Then in the second step, a sparset sub-graph of the graph in the previous step is created. Then in the third step, we introduce a novel graph-based fuzzy C-means clustering approach so that divide users or items into clusters according to the similarity among elements grouped into the same cluster. Then in the fourth step, the prediction engine predicts ratings for items similar to rated ones and finally in the last step, the Top-N recommender, recommends Top-N items. Therefore, this model is known as model-based Fuzzy C-means clustering for CF (in short: MFCCF). In the following subsection, the workflow of the model is described. Fig. 1Open in figure viewerPowerPoint Overview of the MFCCF 3.1 Problem space as a graph We show a graph by 3-tuple G = (N, E, W), while N indicates a set of users or items and E shows the set of edges among the users or items pairs and W indicates the similarity value among every pair. The graph can be created using the information about both users and items. We use Pearson correlation coefficient (PCC) similarity measure to compute the similarity values between each pair of the users in (2) (users p and q) or items in (3) (items i and j) as follows: (2)where is the rating given to item i by the user p, is the average rating given by user i, is the set of items which are rated by both users p and q. (3)where denotes the rate of user p for item i, Pi is the users who rated item i and is the average rating of item i and is defined as follows: (4)where m is the number of users. 3.2 Finding sparset sub-graph For graph G an integer p ≤ |N|, the k-sparsest sub-graph problem searches for a set of p vertices that induces the least number of edges. The modified version of the constraint-based algorithm proposed in [22] is used to find the sparsest sub-graph. The nodes of the sparsest sub-graph are the initial centre set for MFCCF. Thus, the nodes of the sub-graph may have maximum differences with each other. Let G = (N, E) be a directed weighted graph. Given R ⊆ N, the density of the sub-graph R is specified as , where is the weight of edge e. Fig. 2 describes the functionality of the algorithm. This algorithm begins with a considered graph R and calculates the current density . To this end, the nodes with maximum weighted degrees are deleted. Therefore, the node set is decreased to \ , and the algorithm assures that at least one of the sub-graphs under consideration includes approximately nodes [23]. We use this phase to find a subset of a graph as of the sparsest sub-graph to be utilised as an initial centre set for the fuzzy C-means clustering algorithm. Authors in [23] show that one of the intermediate sub-graphs creates a (2 + 2∈) approximation to the sparsest sub-graph. So, the candidate nodes and the threshold value are denoted as and (2 + 2∈) , respectively. Fig. 2Open in figure viewerPowerPoint Approximate sparsest sub-graph detection 3.3 Model-based fuzzy C-means clustering In the model-based fuzzy C-means clustering, we use improved fuzzy C-means as clustering approach to define a variant membership degree of each item that belongs to various clusters (User-item clustering). MFCCF detects a partition of items based on a specified distance measured between two distinct items. The output is the assignment of various items to k different clusters. As a result of the fuzzy C-means method, we obtain a degree of membership of each item to a various cluster. Subsequent to applying the algorithm in Fig. 2, the values of cluster centres for each cluster are detected. We need to discover the number of clusters in which each user occurs. Since we are dealing with soft clusters, we may get a number of cluster numbers for each user. Furthermore, the initial centres of the proposed clustering approach are assigned using the sparsest sub-graph detection algorithm. While the newly found centres are being distinguished, the graph nodes are assigned again to the closest cluster centres. This process will be applied in an iterative manner on all of the clusters up to the centre set does not change. The required parameters are defined in (5) and (6). (5) (6)After each of the iterations, the last centre set is recognised and the other nodes are assigned to their suitable centres. Also, at the end of every iteration, the fuzzy membership is updated using (5). In the end of the fuzzy C-means algorithm if the value of membership is less than the earlier ones, while uij is between 0 and 1; ci is the centre point of cluster i; dij is the Euclidian distance between ith centroid and jth data point and m ∈ [1, ∞] is a weighting exponent. 3.4 Recommendation of MFCCF In the following section, a set of interesting items is recommended to the active users based on the found clusters in the previous step. In order to specify what items to take into account in the prediction step, we apply a fuzzy method. Especially, every item is signed to n clusters with various degrees of membership. Prediction is assessed according to t1 items, which have the maximum value of membership at the cluster1, t2 items, which have the maximum degree of membership at cluster 2, and so on. t1, t2. … tn are chosen in a symmetrical way, e.g. if k = 4 and t = 10 and membership of item i is (0.5,0.3,0.2,0.1), we consider top-9 items from cluster 1, top-3 items from cluster 2 and so on. In the user-item filtering step, we use the Pearson Correlation as a well-defined measure of similarity between two items i and j, while prediction strategies are expressed by (7): (7)where is the rating given to item j by the user p, is the average rating of item i, sim(i, j) is a similarity function between item i and item j that is calculated by (2). 3.5 Top-N collaborative filtering Finally, the top 'n' clusters according to the average value of cluster centre values are chosen. It originates from the basic concept that not all but critical clusters will have a remarkable effect on recommendations. 4 GitHub case study RSs provide filtered information to the users in all fields like e-commerce etc [24]. In this section, we apply our recommender on GitHub to demonstrate that our approach is developing a recommender framework utilising fuzzy C-mean and model-based clustering as Fig. 3. GitHub, a collaborative source code hosting site is based on Git, version control system and has reached a lot of reputation in the recent years. It includes a large amount of data regarding 'Users' and 'Repositories'. Whenever someone wants to contribute on GitHub, he may get confused to which repository they plan to contribute out of huge repositories which may lead to decrease in the count of contributors. In this way, to increase the number of contributors on GitHub, it is needed to produce filtered information to users respecting which repositories to contribute to. RSs are beneficial to resolve this ambiguity among. Fig. 3Open in figure viewerPowerPoint Applying MFCCF on GitHub repositories 4.1 Data acquisition We use GHTorrent [25] project for data acquisition. GHTorrent project is an effort to create a scalable, queriable, offline mirror of data offered through the Github REST API. GHTorrent monitors the GitHub public event timeline. For each event, it retrieves its contents and their dependencies, exhaustively. It then stores the raw JSON responses to a MongoDB database, while also extracting their structure in a MySQL database [26]. In our case study, we need to know about users, projects (in GitHub language: repositories), and projects that users participated in it (in GitHub language: watchers) as Fig. 4. Fig. 4Open in figure viewerPowerPoint Extracted Data from GitHub 4.2 Graph representation We create a graph G = (N, E, W), from extracted data from GitHub, while N indicates a set of users or projects and E shows the set of edges among the users or projects pairs and W indicates the similarity value among every pair. We use Pearson correlation coefficient (PCC) similarity measure to compute the similarity values between each pair of the watchers in (2) (users p and q) or projects in (3) (projects i and j). One-star users can affect the predictive power of our model. Thus, we exclude the users we have little information about them. Our experiments showed that excluding users who rated <30 repositories gives reasonable results. So, after creating a graph from the information in GitHub, we find sparset sub-graph from it using the algorithm in Section 3.2 with a threshold value equal to 30. 4.3 Applying MFCCF Once all information resources are presented in sparset sub-graphs, the experiment took place. 5 Recommendation system assessment Here to evaluate our recommendation methods, we measure its effectiveness regarding the software project similarity measurement and recommending of HRs to project leaders in GitHub as follows. Ten projects are selected. For each project, the project leaders manually selected the most closely related projects. They have no limitation on the number of selected projects. We asked HR managers of the company to select the most fitted HR with the project. The number of HRs assigned to each project depended on its size. Finally, the results of company experts are compared with the results obtained with our recommendation systems. The results are produced by using an arrangement of well-defined quality measure (QM). The comprehensive descriptions of the assessment measures and obtained results are mentioned in the corresponding sections. The experiments have been run on a machine with a Core i5, 2.60 GHz CPU and 4 GB of RAM. Moreover, all of the methods were implemented using Java programming language. With regard to project similarity, the metric precision, recall, and F-measure were calculated. With regard to the assignment of HRs, only the precision was computed because the system was configured to obtain the same number of employees as those selected by the specialists. Various assessment metrics have been utilised to evaluate RSs. These metrics can be classified into accuracy metrics and coverage metrics [27]. The precision measurement measures how well a framework can anticipate a particular rating value for a specific item. There are several accuracy-based metrics in the literature e.g. mean absolute error (MAE), root precision, Pearson product-moment correlation , recall, mean square error (RMSE), and F1 measures [27]. Here Precision, Recall, and F1 measures are utilised to compare the accuracy of the proposed method. The procedure of choosing particular metrics to evaluate the performance of the MFCCF recommendation method depends on the goals that the system is supposed to fulfil. Also, to calculate the Precision, Recall, and F1 measures, it needs to transform the ratings into binary values including relevant and non-relevant items [27]. The Precision measure is defined as a measure of exactness or accuracy. It specifies the ability to retrieve top-ranked documents that are most relevant. This measure is calculated as the fraction of the relevant items retrieved out of all items (see (8)). (8)The Recall is a measure of completeness, which determines the fraction of relevant items retrieved out of all relevant items that were retrieved or not (see (9)). (9)The Precision and Recall measures are obviously conflicting in nature [27]. If the number of recommendations increases, then the number of relevant items and also the Recall measure increases, while at the same time the Precision measure decreases. The F1 measure is utilised to obtain a suitable weighted combination of the Precision and Recall measures (see (10)). (10)The experiment results for project similarity and HR assignment are introduced in the accompanying tables. It ought to be specified that in the project similarity tests, we select three projects in Java language (P11, P12, P13) and Tables 1–3 demonstrate the outcome for project similarity with the P11, P12, and P13, respectively. As the outcomes will be demonstrated noticed, the undertaking with which the best assessment comes about were acquired was P13 with an F-measure score of 0.851. Table 1. Assessment results for project similarity with P11 Project Projects selected manually by expert Projects selected by our RS Projects correctly suggested Precision Recall F-measure 1 13 16 15 0.854 0.800 0.812 2 14 15 14 0.834 0.789 0.793 3 17 18 16 0.810 0.889 0.879 4 15 13 10 0.900 0.870 0.890 5 9 9 8 0.869 0.745 0.823 6 19 20 15 0.756 0.823 0.850 7 22 25 20 0.745 0.810 0.805 8 15 16 15 0.857 0.876 0.903 9 14 13 12 0.887 0.900 0.870 10 12 13 11 0.834 0.835 0.860 average — — — 0.835 0.834 0.849 Table 2. Assessment results for project similarity with P12 Project Projects selected manually by expert Projects selected by our RS Projects correctly suggested Precision Recall F-measure 1 13 14 15 0.862 0.812 0.865 2 14 15 14 0.843 0.779 0.798 3 17 16 14 0.823 0.843 0.870 4 15 16 14 0.897 0.890 0.809 5 9 10 8 0.869 0.756 0.812 6 19 21 17 0.800 0.843 0.846 7 22 21 19 0.845 0.823 0.845 8 15 16 15 0.867 0.800 0.987 9 14 13 11 0.904 0.930 0.813 10 12 13 10 0.823 0.846 0.856 average — — — 0.853 0.832 0.850 Table 3. Assessment results for project similarity with P13 Project Projects selected manually by expert Projects selected by our RS Projects correctly suggested Precision Recall F-measure 1 13 17 15 0.864 0.823 0.812 2 14 13 13 0.854 0.812 0.793 3 17 15 13 0.820 0.898 0.879 4 15 14 14 0.920 0.874 0.890 5 9 12 10 0.869 0.800 0.823 6 19 20 20 0.769 0.790 0.770 7 22 23 19 0.745 0.810 0.835 8 15 14 17 0.877 0.876 0.943 9 14 13 14 0.910 0.900 0.879 10 12 10 13 0.898 0.835 0.889 average — — — 0.853 0.834 0.851 The experiment results for HRs assignment are presented in Tables 4–6 for project P11, P12, and P13. With regard to the assignment of HRs, only the precision was computed because the system was configured to obtain the same number of employees as those selected by the experts. As can be observed, the best results are obtained for P13 with a precision of 0.843. Table 4. Assessment results for HRs assignment to P11 Project HR assignment with leaders HR assignment with our model Precision 1 12 11 0.654 2 14 13 0.634 3 13 11 0.678 4 15 13 0.612 5 10 8 0.534 6 8 6 0.521 7 9 8 0.600 8 14 13 0.634 9 12 11 0.612 10 10 9 0.678 average — — 0.616 Table 5. Assessment results for HRs assignment to P12 Project HR assignment with leaders HR assignment with our model Precision 1 12 10 0.823 2 14 12 0.834 3 13 12 0.865 4 15 14 0.789 5 10 8 0.812 6 8 6 0.825 7 9 6 0.867 8 14 12 0.813 9 12 10 0.816 10 10 8 0.768 average — — 0.821 Table 6. Assessment results for HRs assignment to P13 Project HR assignment with leaders HR assignment with our model Precision 1 12 11 0.876 2 14 13 0.834 3 13 12 0.856 4 15 14 0.877 5 10 10 0.789 6 8 8 0.899 7 9 8 0.800 8 14 13 0.834 9 12 11 0.812 10 10 9 0.854 average — — 0.843 The parameter top_N is an important parameter which can be effective on the Precision and Recall measures. The performed experiments test the effects of various values of the parameter (e.g. = 5, 10, 15, 20, 25, and 30) on the mentioned measures for the GitHub dataset. Table 7 and Fig. 5 report the Precision and Recall analysis for our dataset. The results show that in most cases while the value of the parameter is increased, the Precision value is decreased for all datasets. It can be seen from the results that in most cases when the top_N is increased, the Recall values are also increased, while Fig. 5 shows that under the same conditions, the Precis