## 3 Research Design
### Evaluation Methods
#### 3.4.1 Community Relation Approach  
First, with the community relation approach, we used GitHub's _watching_ and _forking_ features as the ground truth. GitHub users can follow, or watch projects whose developments they want to monitor [39]. If a developer is watching a project, this indicates that he or she is interested in the project. Similarly, _forking_ is used to contribute independently to the project of interest [40]. Developers usually make changes in their forked project (local branch) and can then send their contribution via PRs to base project (master branch). External developers mostly use the fork-pull mechanism to contribute to projects of interest. In this context, we believe that both of features are important for recommendations. While we used \"_watching_ or _forking_\" as an evaluation criteria, we fine-tuned the criteria as detailed below.  
The full name of a GitHub repository is created by concatenating the owner's name (login) with the repository name (e.g., davidteather/handtracking). In analyzing our results, we noticed that the model recommended some project to developers that had only the correct owner login or repository name. In other words, the model suggested an incorrect project of the correct owner or the exact opposite. We evaluated these suggestions as a _half point (0.5)_, as recommending only the correct owner to a developer will still allow the developer to to learn other projects by that owner. Similarly, if the model recommends only a correct repository name with an incorrect owner, this indicates that it has suggested the forked version of a correct repository. Thus, the related developer can discover with the master (base) repository.  
An example scenario demonstrating this situation is given in Table 6. The projects recommended for [PERSON] are listed in the first column Table 6a. Two of them are among the repos that [PERSON] watches, with the two correct matchs, the initial score is 2. There are two repos by a developer named _fengmk2_ among [PERSON]'s watched projects (_fengmk2/parameter_ and _fengmk2/cnpmjs.org_) (Table 6b). The model suggested the project _fengmk2/emoii_ that belongs to a developer who [PERSON] is familiar with him. Similarly, [PERSON] watches a forked project of _visionmedia/co_. Thus, two _half_-score are added to the initial score and _3: (2 + 0.5 + 0.5)_ is the final score.  
We used the Equation (6) to calculate hit score (In other words, the scores of correct recommendations). Our analysis showed that some developers had only a few watched projects. Thus, the case of a developer interested in (watching or forking) fewer than \(n\) projects was considered in the updated score Equation (6).  
\[hit_{score}^{community}=\begin{cases}100*\dfrac{hit_{fullname}+(hit_{ partial}*0.5)}{n},&\text{if }num_{watch\_or\_fork}\geqslant n\\ 100*\dfrac{hit_{fullname}+(hit_{partial}*0.5)}{num_{match\_or\_fork}},&\text{ otherwise}\end{cases} \tag{6}\]  
To sum up, in this evaluation approach, if the recommended project is among the developer's watched or forked projects, the project is considered a hit.  
\begin{table}
\begin{tabular}{l l l} \hline \hline \multicolumn{3}{c}{(a) Recommendations for Alice} \\ \hline Top-5 recommendations & Correct matches & Half matches \\ \hline visionmedia/co & & **co** \\ fengmk2/emoji & & **fengmk2** \\ \hline iojs/io.js & & iojs/io.js \\ julgruber/co-read & & julgruber/co-read \\ koajs/compose/ & & \\ \hline \hline \multicolumn{3}{c}{(b) [PERSON]’s watched repos} \\ \hline Score \({}^{a}\) & Repo full names \\ \hline \hline \multirow{4}{*}{\(\oplus\): correct, \(\oplus\): half, \(\ominus\)wrong.} \\ \cline{2-3}  & & **fengmk2/parameter** \\ \cline{1-1} \cline{2-3}  & & **iojs/io.js** \\ \cline{1-1} \cline{2-3}  & & **julgruber/co-read** \\ \cline{1-1} \cline{2-3}  & & **fengmk2/cnpmjs.org** \\ \cline{1-1} \cline{2-3}  & &... \\ \hline \hline \end{tabular}
\end{table}
Table 6: A sample scenario for recommending with correct and half matches.