A wearable display system comprising: non-transitory computer-readable storage medium configured to store: a neural network for determining a user interface, UI, event trained using: a training image comprising a representation of a training virtual UI device and a training pointer, and a training UI event comprising a state of two or more states of the training virtual UI device based on an interaction of the training pointer with the training virtual UI device in the training image; a display configured to display a virtual UI device at a display location when an image is captured by an image capture device, wherein the virtual UI device is associated with the image at an image location on the image, and wherein the image location is related to the display location; the image capture device configured to capture an image in the physical environment of a user, the image comprising a pointer; and a hardware processor in communication with the image capture device, the display, and the non-transitory computer-readable storage medium, the processor programmed by executable instructions to receive the image comprising the pointer from the image capture device; render a representation of the virtual UI device onto the image at the image location; determine, using the neural network, a UI event with respect to the pointer in the image and the virtual UI device associated with the image; generate virtual content based on the UI event; and cause the display to present the virtual content to the user, wherein the user is wearing the wearable display system. non-transitory computer-readable storage medium configured to store: a neural network for determining a user interface, UI, event trained using: a training image comprising a representation of a training virtual UI device and a training pointer, and a training UI event comprising a state of two or more states of the training virtual UI device based on an interaction of the training pointer with the training virtual UI device in the training image; a training image comprising a representation of a training virtual UI device and a training pointer, and a training UI event comprising a state of two or more states of the training virtual UI device based on an interaction of the training pointer with the training virtual UI device in the training image; a display configured to display a virtual UI device at a display location when an image is captured by an image capture device, wherein the virtual UI device is associated with the image at an image location on the image, and wherein the image location is related to the display location; the image capture device configured to capture an image in the physical environment of a user, the image comprising a pointer; and a hardware processor in communication with the image capture device, the display, and the non-transitory computer-readable storage medium, the processor programmed by executable instructions to receive the image comprising the pointer from the image capture device; render a representation of the virtual UI device onto the image at the image location; determine, using the neural network, a UI event with respect to the pointer in the image and the virtual UI device associated with the image; generate virtual content based on the UI event; and cause the display to present the virtual content to the user, wherein the user is wearing the wearable display system. receive the image comprising the pointer from the image capture device; render a representation of the virtual UI device onto the image at the image location; determine, using the neural network, a UI event with respect to the pointer in the image and the virtual UI device associated with the image; generate virtual content based on the UI event; and cause the display to present the virtual content to the user, wherein the user is wearing the wearable display system.