A system for remotely controlling extended reality by a virtual mouse, comprising: a head mounted display, comprising an image capture device; and a first remote controller for being worn on a hand, wherein the first remote controller communicatively connects to the head mounted display, wherein the first remote controller comprises a first inertial measurement unit, wherein the head mounted display is configured to: capture (S101, S301) an image through the image capture device; detect the image to obtain first data of the hand in the image and second data of a plane in the image; enable the virtual mouse according to the first data and the second data; receive a first signal (61-64; 71, 73, 75, 77, 78; 81, 82) generated by the first inertial measurement unit; perform a click operation by the virtual mouse in an extended reality scene according to the first signal (61-64; 71, 73, 75, 77, 78; 81, 82); detect a first occurring wave peak and a first occurring wave valley of the first signal (61-64; 71, 73, 75, 77, 78; 81, 82); in response to a first time point of the first wave peak being later than a second time point of the first wave valley, perform a first scroll operation by the virtual mouse in an extended reality scene; and in response to the first time point of the first wave peak being earlier than the second time point of the first wave valley, perform a second scroll operation by the virtual mouse in the extended reality scene. a head mounted display, comprising an image capture device; and a first remote controller for being worn on a hand, wherein the first remote controller communicatively connects to the head mounted display, wherein the first remote controller comprises a first inertial measurement unit, wherein the head mounted display is configured to: capture (S101, S301) an image through the image capture device; detect the image to obtain first data of the hand in the image and second data of a plane in the image; enable the virtual mouse according to the first data and the second data; receive a first signal (61-64; 71, 73, 75, 77, 78; 81, 82) generated by the first inertial measurement unit; perform a click operation by the virtual mouse in an extended reality scene according to the first signal (61-64; 71, 73, 75, 77, 78; 81, 82); detect a first occurring wave peak and a first occurring wave valley of the first signal (61-64; 71, 73, 75, 77, 78; 81, 82); in response to a first time point of the first wave peak being later than a second time point of the first wave valley, perform a first scroll operation by the virtual mouse in an extended reality scene; and in response to the first time point of the first wave peak being earlier than the second time point of the first wave valley, perform a second scroll operation by the virtual mouse in the extended reality scene. capture (S101, S301) an image through the image capture device; detect the image to obtain first data of the hand in the image and second data of a plane in the image; enable the virtual mouse according to the first data and the second data; receive a first signal (61-64; 71, 73, 75, 77, 78; 81, 82) generated by the first inertial measurement unit; perform a click operation by the virtual mouse in an extended reality scene according to the first signal (61-64; 71, 73, 75, 77, 78; 81, 82); detect a first occurring wave peak and a first occurring wave valley of the first signal (61-64; 71, 73, 75, 77, 78; 81, 82); in response to a first time point of the first wave peak being later than a second time point of the first wave valley, perform a first scroll operation by the virtual mouse in an extended reality scene; and in response to the first time point of the first wave peak being earlier than the second time point of the first wave valley, perform a second scroll operation by the virtual mouse in the extended reality scene.