# A Data-Driven Game Theoretic Strategy for Developers in Software Crowdsourcing: A Case Study
## 3 Research Design
### Methodology
#### 3.2.1 Build a Model of Crowdsourcing Ability Assessment for Crowd Workers  
For the crowd workers, the process of crowdsourcing software development is to find the task distributed by the platform, register the task, submit programs, review of programs, publish the notice, and collect the reward. In the process, the worker's work of each stage can directly determine whether he or she can win the monetary reward in the end. In order to build a model of crowdsourcing ability assessment for workers, we collected the worker's information at different stages. According to the development process, a worker's crowdsourcing ability mainly shows in three aspects: 1) the ratio of true participation in the crowdsourcing task, 2) the overall quality of the submissions, and 3) the worker's overall ranking. Before building the model, we needed to define some indicators. **Definition 1**. : _Ratio of true participation, an indicator to measure the true involvement of a worker in crowdsourcing development, rather than merely registering tasks and being an observer. The ratio of true participation is calculated as below:_  
\[T_{p}\big{(}w_{i},v_{j}\big{)}=SR\big{(}w_{i},v_{j}\big{)}\times S\big{(}w_{i},v_{j}\big{)} \tag{1}\]  
In Equation (1), \(SR\big{(}w_{i},v_{j}\big{)}\) stands for the ratio of solutions submitted to crowdsourcing tasks of different types (generally the crowdsourcing tasks fall into four types: Code, F2F, Assembly, Architecture) of the worker \(i\), where the ratio of true participation is in direct proportion to submission ratio and times of participation, which can not only measure the worker's attention paid to the crowdsourcing tasks, but also ensure that his participation times are sufficient. **Definition 2**. : _The overall quality of the submissions, the main purpose of which is to judge whether the crowdsourcing tasks are completed in good quality. The specific calculation is shown in Equation (2)._  
\[S_{q}\big{(}w_{i},v_{j}\big{)}=PR\big{(}w_{i},v_{j}\big{)}\times RR\big{(}w_{i },v_{j}\big{)} \tag{2}\]  
In Equation (2), \(RR\big{(}w_{i},v_{j}\big{)}\) stands for the ratio of the worker \(i\)'s submissions that have passed the review when he/she takes part in tasks of certain types. We can see that the overall quality is the product of the number of submissions that have passed the review, multiplying the ratio of submissions passed the review. The use of product was to avoid the consequence that a single attribute may have a great effect on the results. **Definition 3**. : _The overall ranking. It can, most directly, show the worker's winning situation in a certain type of crowdsourcing task, and indirectly reflect the worker's overall income. The calculation is shown as below:_  
\[O_{r}\big{(}w_{i},v_{j}\big{)}=WP\big{(}w_{i},v_{j}\big{)}\times W\big{(}w_{i },v_{j}\big{)} \tag{3}\]  
In Equation (3), \(WP\big{(}w_{i},v_{j}\big{)}\) represents the proportion of the crowd worker's submissions, from the passage of the review to winning the final. \(W\big{(}w_{i},v_{j}\big{)}\) represents the number of times that the worker has won the crowdsourcing tasks. After some indicators were well defined, the model of crowdsourcing ability assessment was calculated in detail, as below:  
\[CA\big{(}w_{i},v_{j}\big{)}=\frac{\sqrt{Tp\big{(}w_{i},v_{j}\big{)}^{2}+Sq \big{(}w_{i},v_{j}\big{)}^{2}+Or\big{(}w_{i},v_{j}\big{)}^{2}}}{Ap\big{(}w_{i },v_{j}\big{)}+\mu\big{(}w_{i},v_{j}\big{)}} \tag{4}\]  
In Equation (4), \(Ap\big{(}w_{i},v_{j}\big{)}\) is the worker's average ranking. In the process of building the model, we found out that the average ranking of some workers had been lost. In order to eliminate the assessment inaccuracy caused by the loss of average rankings, a parameter \(\mu\big{(}w_{i},v_{j}\big{)}\) was employed.