# Review of Code Similarity and Plagiarism Detection Research Studies
## 3 Code Similarity Detection Techniques
### Token-Based Techniques  
Token-based techniques regard code as a sequence of tokens during the transformation and detection steps. After tokens are extracted from the code using lexical analysis, the way this information is used can vary significantly depending on the nature of the technique. Token-based techniques follow a method of finding matching sub-sequences in the entire token sequence and outputting the corresponding code parts as a code clone. Because codes are converted into token sequences, token-based techniques have an advantage in that they are not sensitive to minor changes, such as differences in code style, unlike text-based techniques, and have a high possibility of detecting meaningful similar codes because they include lexical analysis. However, because lexical analysis is required, token-based techniques cannot be applied completely independently of language, unlike text-based techniques, although it is still easier for token-based techniques to support new languages than it is for tree-based techniques.  
One of the most famous token-based techniques is _CCFinder_[16], which supports a variety of programming languages, such as Java, C/C++, COBOL, VB, and C#. A new version, _CCFinderX_[17], was recently released [47]. CCFinder uses a lexical analyzer to extract tokens from each line of code and integrates tokens from the entire code into a sequence of tokens. Thereafter, the tokens are converted in two ways. The first is to modify the tokens according to the rules defined for each supported language. This is carried out to discriminate between codes with different grammatical structures but similar meanings. The second is a transformation that replaces identifiers corresponding to variables, functions, and type names with special tokens. The suffix tree-matching algorithm [48] is then used to find matches in this transformed token sequence.  
[PERSON]'s techniques [10; 13; 46], which were introduced in Section 3.2, can also be classified as token-based techniques because they use lexical analysis. In particular, [PERSON] applied a parameterized matching technique that replaces the same identifier with the same fully qualified name, as described in Section 2.2. For example, code snippets max(a,b,a) and min(a,a,b) on different lines can be converted into method0(var0, var1, var0) and method0(var0, var0, var1), respectively. Using this transformation method, code clones can be detected by distinguishing between different parameter sequences, etc.  
_CP-Miner_[18; 19] uses a frequent subsequence mining algorithm called CloSpan [49] to find matching token sequences differently from the previously introduced techniques. A _closed subsequence_ implies that the support of the subsequence is different from that of the sequence containing it, which CloSpan efficiently finds. CP-Miner uses extended CloSpan to detect gap clones without being affected by the addition or deletion of sentences. This overcomes the limitations of CCFinder and [PERSON]'s techniques, which are affected by the order in which tokens appear. CP-Miner has succeeded in detecting more than 150,000 clone pairs in a large-scale codebase, demonstrating that it can efficiently detect copy-pasted code, given a large amount of code.  
Token-based techniques are also widely used in plagiarism detection because of their ease of supporting multiple languages and detecting code clones in multiple files. Among the techniques discussed in Section 4, those that employ token-based methods include _Winnowing_[50] (used by _MOSS_), _JPlag_[20], and _SIM_[21]. In the case of Winnowing, its method of using the fingerprint of the code in plagiarism detection is less affected by the partial addition, deletion, and modification of code; however, the fundamental limitations of token-based techniques still exist.