An endoscope observation assistance system comprising: a three-dimensional medical image formation unit that forms a three-dimensional medical image representing the body cavity of a subject to be examined; a treatment tool position detection unit that detects, in real time, the position of a treatment tool inserted into the body cavity, wherein the treatment tool position detection unit detects a direction in which the treatment tool moves straight forward at the same time as detection of the position of the treatment tool; a position-of-interest specifying unit that specifies, as a position of interest, the position of a structure of interest in the body cavity in the three-dimensional medical image; a virtual endoscope image generation unit that receives the three-dimensional medical image as an input, and generates a virtual endoscope image representing the body cavity viewed from a corresponding treatment tool position, which is a position in the three-dimensional medical image corresponding to the detected position of the treatment tool; and a display unit that displays the virtual endoscope image, wherein the virtual endoscope image generation unit generates the virtual endoscope image, the field of view of which includes the specified position of interest and wherein the virtual endoscope image generation unit generates the virtual endoscope image in which image information on a treatment tool visual line from the corresponding treatment tool position is projected, and the treatment tool visual line having a direction in the three-dimensional medical image corresponding to the direction in which the treatment tool moves straight forward. a three-dimensional medical image formation unit that forms a three-dimensional medical image representing the body cavity of a subject to be examined; a treatment tool position detection unit that detects, in real time, the position of a treatment tool inserted into the body cavity, wherein the treatment tool position detection unit detects a direction in which the treatment tool moves straight forward at the same time as detection of the position of the treatment tool; a position-of-interest specifying unit that specifies, as a position of interest, the position of a structure of interest in the body cavity in the three-dimensional medical image; a virtual endoscope image generation unit that receives the three-dimensional medical image as an input, and generates a virtual endoscope image representing the body cavity viewed from a corresponding treatment tool position, which is a position in the three-dimensional medical image corresponding to the detected position of the treatment tool; and a display unit that displays the virtual endoscope image, wherein the virtual endoscope image generation unit generates the virtual endoscope image, the field of view of which includes the specified position of interest and wherein the virtual endoscope image generation unit generates the virtual endoscope image in which image information on a treatment tool visual line from the corresponding treatment tool position is projected, and the treatment tool visual line having a direction in the three-dimensional medical image corresponding to the direction in which the treatment tool moves straight forward.