A manipulator system comprising: a master manipulator configured to send an input command based on manipulation by an operator; a slave manipulator configured to operate according to the input command; an image capturing unit configured to acquire an image of an object; a display device placed in front of the operator and configured to display the image acquired by the image capturing unit; a detection means configured to detect a direction of an operator's face with respect to the display device; and a control unit (31, 32) configured to determine whether the direction of the face is within a predetermined angle with respect to the display device based on a detection result in the detection means, and to shift an operation mode of the slave manipulator between a predetermined first control mode and a second control mode in which an operation is limited more than in the first control mode based on a determination result, characterized in that the detection means comprises: an index unit that is mountable on the operator to be moved along with the operator's face; and a detection unit that is positioned with respect to the display device and that is configured to detect the index unit within a predetermined range, wherein the control unit (31, 32) is configured to normally operate the slave manipulator in the second control mode when the index unit is not detected within the predetermined range in the detection unit, to shift to an input standby mode of an approval signal by the operator to perform a shift from the second control mode to the first control mode when a state in which the index unit is not detected within the predetermined range is changed into a state in which the index unit is detected within the predetermined range, and to perform the shift from the second control mode to the first control mode when the approval signal is input, wherein the control unit (31, 32) is configured to perform the shift from the second control mode to the first control mode without input of the approval when the direction of the face is within the range of the predetermined angle with respect to the display device while an operation amount of the master manipulator from when the shift from the first control mode to the second control mode is performed is within a predetermined operation amount. a master manipulator configured to send an input command based on manipulation by an operator; a slave manipulator configured to operate according to the input command; an image capturing unit configured to acquire an image of an object; a display device placed in front of the operator and configured to display the image acquired by the image capturing unit; a detection means configured to detect a direction of an operator's face with respect to the display device; and a control unit (31, 32) configured to determine whether the direction of the face is within a predetermined angle with respect to the display device based on a detection result in the detection means, and to shift an operation mode of the slave manipulator between a predetermined first control mode and a second control mode in which an operation is limited more than in the first control mode based on a determination result, characterized in that an index unit that is mountable on the operator to be moved along with the operator's face; and a detection unit that is positioned with respect to the display device and that is configured to detect the index unit within a predetermined range, wherein the control unit (31, 32) is configured to normally operate the slave manipulator in the second control mode when the index unit is not detected within the predetermined range in the detection unit, to shift to an input standby mode of an approval signal by the operator to perform a shift from the second control mode to the first control mode when a state in which the index unit is not detected within the predetermined range is changed into a state in which the index unit is detected within the predetermined range, and to perform the shift from the second control mode to the first control mode when the approval signal is input, wherein the control unit (31, 32) is configured to perform the shift from the second control mode to the first control mode without input of the approval when the direction of the face is within the range of the predetermined angle with respect to the display device while an operation amount of the master manipulator from when the shift from the first control mode to the second control mode is performed is within a predetermined operation amount.