A computer-implemented method for controlling a virtual environment, comprising: recognizing a virtual skeleton that models a human subject with a plurality of virtual joints defined by three-dimensional positions, the virtual skeleton derived from depth video of a depth camera imaging the human subject; using the virtual skeleton to identify a gesture performed by the human subject; and controlling a property of the virtual environment as a function of the gesture and a magnitude of the gesture; characterized in that the method further comprises: recognizing an optical flow of the depth video, the optical flow derived from a plurality of cells mapped to the depth video, each cell having a motion value indicating a magnitude of motion in that cell recognized between consecutive image frames (210, 212) of the depth video; using the optical flow to assign the magnitude to the gesture performed by the human subject; generating an energy field (214; 508; 602) created by motion of the human subject as a function of the optical flow and specified rules of a physical simulation of the virtual environment, the energy field including a plurality of energy elements derived from the plurality of cells of the optical flow, each energy element including a plurality of energy properties derived from the motion value of a corresponding cell and the specified rules of the physical simulation; and mapping the energy field to a virtual position in the virtual environment so that the energy field may interact with virtual objects in the virtual environment, wherein the mapping comprises one or more scaling/translation/rotation transformations applied according to the specified rules of a physical simulation of the virtual environment; wherein the controlling comprises controlling the property of the virtual environment as a function of the gesture, the energy field, and the magnitude of the gesture. recognizing a virtual skeleton that models a human subject with a plurality of virtual joints defined by three-dimensional positions, the virtual skeleton derived from depth video of a depth camera imaging the human subject; using the virtual skeleton to identify a gesture performed by the human subject; and controlling a property of the virtual environment as a function of the gesture and a magnitude of the gesture; characterized in that the method further comprises: recognizing an optical flow of the depth video, the optical flow derived from a plurality of cells mapped to the depth video, each cell having a motion value indicating a magnitude of motion in that cell recognized between consecutive image frames (210, 212) of the depth video; using the optical flow to assign the magnitude to the gesture performed by the human subject; generating an energy field (214; 508; 602) created by motion of the human subject as a function of the optical flow and specified rules of a physical simulation of the virtual environment, the energy field including a plurality of energy elements derived from the plurality of cells of the optical flow, each energy element including a plurality of energy properties derived from the motion value of a corresponding cell and the specified rules of the physical simulation; and mapping the energy field to a virtual position in the virtual environment so that the energy field may interact with virtual objects in the virtual environment, wherein the mapping comprises one or more scaling/translation/rotation transformations applied according to the specified rules of a physical simulation of the virtual environment; recognizing an optical flow of the depth video, the optical flow derived from a plurality of cells mapped to the depth video, each cell having a motion value indicating a magnitude of motion in that cell recognized between consecutive image frames (210, 212) of the depth video; using the optical flow to assign the magnitude to the gesture performed by the human subject; generating an energy field (214; 508; 602) created by motion of the human subject as a function of the optical flow and specified rules of a physical simulation of the virtual environment, the energy field including a plurality of energy elements derived from the plurality of cells of the optical flow, each energy element including a plurality of energy properties derived from the motion value of a corresponding cell and the specified rules of the physical simulation; and mapping the energy field to a virtual position in the virtual environment so that the energy field may interact with virtual objects in the virtual environment, wherein the mapping comprises one or more scaling/translation/rotation transformations applied according to the specified rules of a physical simulation of the virtual environment; wherein the controlling comprises controlling the property of the virtual environment as a function of the gesture, the energy field, and the magnitude of the gesture.