# A System Design Perspective for Business Growth in a Crowdsourced Data Labeling Practice
## 3 Payment Mechanism
### Credit Function Payment  
We review the original skip-based formula of [10] before describing the modifications proposed to the credit function of the labeling system. This formula is used to define the basic version of our credit function. Table 2 describes the notation used together with the corresponding definitions and mathematical domains. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Symbol** & **Definition** & **Domain** \\ \hline \(N\) & Total number of questions & \(\mathbb{Z}^{+}\) \\ \(G\) & Total number of golden questions & \([0,N]\in\mathbb{Z}^{+}\) \\ \(i\) & Counter of golden questions IDs & \(\{1,\dots,G\}\) \\ \(t\) & indexed from 1 through \(G\) & \(\{+,-\}\) \\ \(s\) & Possible response of users to golden items & \(\{Correct,\,Incorrect,\,Skip\}\) \\  & Maximum credit paid when providing & \(\mathbb{R}^{+}\) \\ \(\mu_{\text{max}}\) & correct answers to all golden questions & \([0,\mu_{\text{max}})\) \\ \(\mu_{\text{min}}\) & Minimum credit paid & \([0,1]\) \\  & in terms of users’ participation & \\ \(T\) & Confidence level or shape parameter & \((0,1]\) \\  & of the credit function & \\ \(q_{i}^{s}\) & Response state to the \(i\)th golden item & \(\left\{\begin{array}{ll}1,&\text{if }ith\text{ golden item is answered via state }s\\ 0,&\text{otherwise}\\ \end{array}\right.\) \\ \(\lambda_{i}^{t}\) & Type of \(i\)th golden question & \(\left\{\begin{array}{ll}1,&\text{if }ith\text{ golden item is }type\ t\\ 0,&\text{otherwise}\\ \end{array}\right.\) \\ \(Bonus_{s}^{t}\) & Coefficient assigned to a response from \(S\) & \(\left\{\begin{array}{ll}Bonus_{\text{correct}}^{+},Bonus_{\text{skip}}^{+},Bonus_{\text{incorrect}}^{+}\\ Bonus_{\text{correct}},Bonus_{\text{skip}}^{+},Bonus_{\text{incorrect}}\\ \end{array}\right.\) \\ \(g_{s}^{t}\) & Number of type \(t\) golden items & \([0,G]\) \\  & assigned state \(s\) as answer & \\  & Number of unspecified golden items & \\ \(g_{s}^{\circ}\) & (according to the original formulation setting) & \([0,G]\) \\  & assigned state \(s\) as answer & \\  & Coefficient assigned to a correct answer & \\ \(Bonus_{\text{correct}}^{\circ}\) & given to an unspecified golden question & \(\frac{1}{T}\) \\  & (according to the original formulation setting) & \(0\) \\ \(Bonus_{\text{incorrect}}^{\circ}\) & given to a positive-type golden question, & \(\frac{1}{T}\) \\  & e.g., a True-sign (\(\blacktriangledown\)) assignment & \\  & to the celebrity image detection & \\ \(Bonus_{\text{skip}}^{+,-,\circ}\) & Coefficient assigned to skipping & \\  & any type of golden question & \(1\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Notations of the payment function. \[(\mu_{\max}-\mu_{\min})T^{G}\prod_{i=1}^{G}\left(\sum_{s}q_{i}^{s} Bonus_{s}^{\circ}\right)+\mu_{\min} \tag{1}\] \[=(\mu_{\max}-\mu_{\min})T^{G}\left\{(Bonus_{Correct}^{\circ})^{S_{ Correct}^{S_{Correct}}}(Bonus_{Incorrect}^{S_{Incorrect}})^{S_{Incorrect}^{S_{ higher}}}\left(Bonus_{Skip}^{\circ}\right)^{S_{Skip}}\right\}+\mu_{\min}\]  
The original formulation of the payment function obeys Relation (1), in which the values of the incorrect, skip, and correct coefficients are equal to 0, 1, and \(\frac{1}{7}\), respectively. Note that the type of golden questions is omitted from the computation, which results in only three coefficients: \(Bonus_{Correct}^{\circ}\), \(Bonus_{Incorrect}^{\circ}\), and \(Bonus_{Skip}^{\circ}\). An underlying assumption is \(\sum\limits_{s}q_{i}^{s}=1,\forall i\), which imposes a single label per question. The extensive form of the formula is completed through the \(g_{Correct}^{\circ}\), \(g_{Incorrect}^{\circ}\), and \(g_{Skip}^{\circ}\) exponents included within the multiplication of the corresponding coefficients. The reasoning behind the choice of \(\frac{1}{7}\) is to give the maximum credit to an individual assigning correct labels to all the golden questions. If a contributor responds to all the golden questions correctly, the formula becomes \((\mu_{\max}-\mu_{\min})T^{G}\left(\frac{1}{7}\right)^{G}+\mu_{\min}\), which yields \(\mu_{\max}\).