## 1 Introduction  
Malicious tampering and unauthorized usage of software are serious concerns for the computing industry. According to a recent study, unauthorized usage of software costs nearly $400 billion in 2015 [1]. Code obfuscation technique based on process-level virtual machines (PVMs) is becoming a viable means to protect software systems from malicious tampering. Recent studies have shown that such techniques are effective in managing digital copy rights [2], code protection [3], and tamper-proofing [4]. With PVM-based tamper-proofing techniques, the native codes of the crucial part of the software are transformed into bespoke virtual machine instructions (bytecodes). These bytecodes will then be translated into machine code by a virtual machine (VM) interpreter during runtime. The underlying principal of PVM-based protection is to force the attacker to move from a familiar computingenvironment to the unfamiliar and obfuscated virtual environment, which can significantly increase the cost of software tampering so that it prevents tampering and deters unauthorized reverse engineering of software. The VM system plays a key role in any PVM-based protection scheme because it determines how the virtual instructions are scheduled and interpreted--which ultimately determines which information and software behavior will be exposed to the user and attackers. Therefore, it is crucially important to protect the VM against malicious tampering. Prior approaches on PVM-based code protection work by creating a new section in the program binary to store virtual instructions of protected code regions. However, the new section is vulnerable to advanced attacks where an adversary can use sophisticated tools (e.g., Ollydbg and IDA Pro) to locate the section so as to discover the information of VM implementation and virtual instruction sets. Holding such information helps an adversary better understand the working mechanism of the VM. As a result, the security of the protected code regions will be greatly compromised. This paper presents _VMGuards_, a novel tampering detection system. Contrary to previous work, our approach regards the security of the VM as the first class concern in the system design. We achieve this by providing two new sets of virtual instructions to protect the internal implementation of the VM and the host runtime environment where the VM runs in from code tampering. Firstly, we design a set of Tamper Proofing Instructions (TPIs) to protect the internal code integrity. Our TPIs extend the standard virtual instruction by providing additional information for the VM Interpreter to check the starting and ending addresses of the protected code region, as well as the checksum of the code to prevent code from unauthorized modifications. Our careful design ensures that TPIs have the same structure as the standard virtual instructions so as not to cause the attackers' suspicions. Secondly, we provide a set of Anti-Debug Instructions (ADIs) to protect the host runtime environment (e.g., operating systems and libraries) where the VM runs in against code tampering. These two instruction sets, put together, allow us to build a code protection system that is stronger than any previous work we have seen so far. By carefully integrating the instructions with the VM implementation, we are able to build a code protection with little extra overhead compared to existing solutions. We evaluated _VMGuards_ using a set of real-life applications. Experimental results show that _VMGuards_ provides stronger protection than the state-of-the-art code protection systems at the same overhead. This paper makes the following specific contributions:  
1. It is the first work on PVM-based code protection that provides security protection for the VM system. 2. It introduces a novel set of tamper-proofing instructions to protect critical internal implementation from runtime code tampering. 3. It provides a new set of anti-debugging instructions to detect the presence of code tampering in the current host environment.