The answer to this question depends on the software itself, but most antivirus tools look for file signatures first. The next step is generally looking for files that are “relatives” of known malware. The last step is examining each file more closely by either running it in a sandbox or through machine learning algorithms.