Patent Description:
A display, such as an augmented reality (AR) device that displays a state where an image is superimposed on a real world, is known in the related art. A technique that reproduces a display state displayed on the augmented reality device with another display device is known.

For example, <CIT> discloses a technique for recording an image captured by a camera and a displayed virtual image and causing another display device to display the image and the virtual image according to user's designation in a display system that generates a virtual image corresponding to augmented reality (AR) and causes a display unit to display the virtual image. Image processing is also disclosed in <CIT> and <CIT>.

According to the technique disclosed in <CIT>, the degree of reproduction may not be sufficient in a case where a virtual image displayed on a transmission-type display is reproduced with a non-transmission-type display.

The present disclosure has been made in consideration of the above-mentioned circumstances, and an object of the present disclosure is to provide an image processing device, an image processing method, and an image processing program that can accurately reproduce a visually recognized state shown on a transmission-type display, as defined by the claims.

According to the present disclosure, a visually recognized state shown on a transmission-type display can be accurately reproduced.

Examples of an embodiment of a technique of the present disclosure will be described in detail below with reference to the drawings.

The configuration of an image display system <NUM> of the present embodiment will be described with reference to <FIG>. As shown in <FIG>, the image display system <NUM> of the present embodiment comprises a glasses-type information display device <NUM>, an image processing device <NUM>, a smartphone <NUM>, and a printer <NUM>. The glasses-type information display device <NUM>, the image processing device <NUM>, the smartphone <NUM>, and the printer <NUM> are connected to each other via a network <NUM> by wired communication or wireless communication.

The glasses-type information display device <NUM> projects a projection image, which is provided from a smartphone <NUM>, to a user who visually recognizes a real image using AR glasses <NUM>, so that the user can visually recognize a state where a virtual image is superimposed on a visual field of a real image.

The image processing device <NUM> generates an image, which simulates the transmission state of a virtual image via the glasses-type information display device <NUM>, on the basis of parameters representing the characteristics of the model of the glasses-type information display device <NUM> from a projection image that serves as the basis of the virtual image and a captured image that indicates a real image. Hereinafter, an image that simulates the transmission state of the virtual image via the glasses-type information display device <NUM> will be referred to as a composite image.

The smartphone <NUM> can acquire the composite image, which is generated by the image processing device <NUM>, to display a visually-recognized image, which is visually recognized by the user using the AR glasses <NUM> of the glasses-type information display device <NUM> and in which the real image and the virtual image are combined with each other, using a display 16A. Further, the printer <NUM> can acquire the composite image, which is generated by the image processing device <NUM>, to print the visually-recognized image which is visually recognized by the user using the AR glasses <NUM> of the glasses-type information display device <NUM> and in which the real image and the virtual image are combined with each other.

The configuration of the glasses-type information display device <NUM> of the present embodiment will be described with reference to <FIG>. As shown in <FIG>, the glasses-type information display device <NUM> of the present embodiment comprises augmented reality (AR) glasses <NUM> and a smartphone <NUM>.

The AR glasses <NUM> are a device that allows a user to visually recognize a projection image, which is projected from an organic light emitting diode (OLED) <NUM>, in a state where the projection image is superimposed on a real image. <FIG> is a perspective view of an example of the AR glasses <NUM> of the present embodiment. As shown in <FIG> and <FIG>, the AR glasses <NUM> comprise a pair of a transmission unit <NUM> for a left eye and a transmission unit 20R for a right eye, an OLED <NUM>, a camera <NUM>, and a space recognition sensor <NUM>. The transmission unit 20R for a right eye of the present embodiment is an example of a transmission unit of the present disclosure.

The OLED <NUM> projects an image (projection image), which represents information, onto the transmission unit 20R for a right eye in order to insert information into the visual field of a real image, which is visually recognized by the user through the transmission unit 20R for a right eye, in a superimposed manner. The OLED <NUM> of the present embodiment is an example of a projection unit of the present disclosure, and the projection image projected from the OLED <NUM> is an example of a first image of the present disclosure.

The transmission unit 20R for a right eye includes a lens 22R for a right eye and a light guide plate <NUM>. Light corresponding to the projection image projected from the OLED <NUM> is incident on one end of the light guide plate <NUM>. The direction of light propagated through the light guide plate <NUM> is changed at an emission portion (not shown), and the light is emitted in a direction of the user's eye. The light, which is emitted from the light guide plate <NUM> and corresponds to the projection image, is transmitted through the lens 22R for a right eye and is guided to the right eye of the user. Further, the user visually recognizes the real world, which is shown through the lens 22R for a right eye, as a real image with the right eye.

For this reason, while the projection image is projected from the OLED <NUM>, the visually-recognized image visually recognized with the right eye of the user is in a state where the projection image projected onto the light guide plate <NUM> is superimposed on the real image representing the real world shown through the lens 22R for a right eye. Further, while the projection image is not projected from the OLED <NUM>, the visually-recognized image visually recognized by the user is the real image that represents the real world shown through the lens 22R for a right eye and the light guide plate <NUM>.

The camera <NUM> is a camera that images the real world visually recognized by a user. Examples of the camera <NUM> include a digital camera, such as a complementary metal oxide semiconductor (CMOS) camera, and it is preferable that a color image can be captured. Image data of a captured image, which is captured by the camera <NUM>, are output to the smartphone <NUM>. The captured image, which is captured by the camera <NUM> of the present embodiment and indicates the real image, is an example of a second image of the present disclosure.

The space recognition sensor <NUM> is a sensor that detects a distance to a subject present in the real world visually recognized by a user. Examples of the space recognition sensor <NUM> include a monocular camera, a stereo camera, a TOF camera, and the like. A detection result of the space recognition sensor <NUM> is output to the smartphone <NUM>. Since the detection result of the space recognition sensor <NUM> is data indicating the position of the subject present in the real world, the detection result of the space recognition sensor <NUM> is referred to as "position data" hereinafter.

While the projection image is projected from the OLED <NUM>, a captured image is captured by the camera <NUM> and position data are acquired by the space recognition sensor <NUM>. The image data of the captured image and the position data of the space recognition sensor <NUM> are stored in the smartphone <NUM> in association with the image data of the projected projection image.

Further, the transmission unit <NUM> for a left eye includes a lens <NUM> for a left eye. The user visually recognizes the real world, which is shown through the lens <NUM> for a left eye, with the left eye.

Meanwhile, the smartphone <NUM> comprises a processor <NUM>. The processor <NUM> of the present embodiment controls the OLED <NUM> to project the projection image onto the light guide plate <NUM> from the OLED <NUM>. Further, while the OLED <NUM> projects the projection image onto the light guide plate <NUM>, the processor <NUM> controls the camera <NUM> to image the real world visually recognized by the user and controls the space recognition sensor <NUM> to detect a distance to a subject present in the real world visually recognized by the user.

<FIG> is a block diagram showing an example of the configuration of the smartphone <NUM> that is related to a function related to the projection of the projection image. As shown in <FIG>, the smartphone <NUM> comprises a processor <NUM>, a memory <NUM>, an interface (I/F) unit <NUM>, a storage unit <NUM>, a display <NUM>, and an input device <NUM>. The processor <NUM>, the memory <NUM>, the I/F unit <NUM>, the storage unit <NUM>, the display <NUM>, and the input device <NUM> are connected to each other via a bus <NUM>, such as a system bus or a control bus, such that various types of information can be given and received therebetween.

The processor <NUM> reads out various programs, which include a display control program <NUM> stored in the storage unit <NUM>, to the memory <NUM> and performs processing corresponding to the program read out. Accordingly, the processor <NUM> controls the projection of the projection image that is performed by the OLED <NUM>. The memory <NUM> is a work memory that is used in a case where the processor <NUM> performs processing.

The display control program <NUM>, the image data (not shown) of the projection image projected from the OLED <NUM>, various other types of information, and the like are stored in the storage unit <NUM>. Specific examples of the storage unit <NUM> include a hard disk drive (HDD), a solid state drive (SSD), and the like.

The I/F unit <NUM> communicates various types of information to each of the OLED <NUM>, the camera <NUM>, the space recognition sensor <NUM>, and the image processing device <NUM> using wireless communication or wired communication. The display <NUM> and the input device <NUM> function as a user interface. The display <NUM> provides various types of information, which is related to the projection of the projection image, to a user. The display <NUM> is not particularly limited, and examples of the display <NUM> include a liquid crystal monitor, a light emitting diode (LED) monitor, and the like. Further, the input device <NUM> is operated by a user so that various instructions related to the projection of the projection image are input. The input device <NUM> is not particularly limited, and examples of the input device <NUM> include a keyboard, a touch pen, a mouse, and the like. A touch panel display in which the display <NUM> and the input device <NUM> are integrated with each other is employed in the smartphone <NUM>.

Meanwhile, the image processing device <NUM> has a function of generating a composite image that simulates the transmission state of the virtual image via the glasses-type information display device <NUM> as described above.

<FIG> is a block diagram showing an example of the configuration of the image processing device <NUM> that is related to a function related to the generation of the composite image. As shown in <FIG>, the image processing device <NUM> comprises a processor <NUM>, a memory <NUM>, an I/F unit <NUM>, a storage unit <NUM>, a display <NUM>, and an input device <NUM>. The processor <NUM>, the memory <NUM>, the I/F unit <NUM>, the storage unit <NUM>, the display <NUM>, and the input device <NUM> are connected to each other via a bus <NUM>, such as a system bus or a control bus, such that various types of information can be given and received therebetween.

The processor <NUM> reads out various programs, which include an image processing program <NUM> stored in the storage unit <NUM>, to the memory <NUM> and performs processing corresponding to the program read out. Accordingly, the processor <NUM> performs processing for generating the composite image. The memory <NUM> is a work memory that is used in a case where the processor <NUM> performs processing.

The image processing program <NUM>, the image data of the projection image projected from the OLED <NUM> and the image data of the captured image captured by the camera <NUM> in the glasses-type information display device <NUM>, various other types of information, and the like are stored in the storage unit <NUM>. Specific examples of the storage unit <NUM> include a HDD, an SSD, and the like.

The I/F unit <NUM> communicates various types of information to each of the smartphone <NUM>, the smartphone <NUM>, and the printer <NUM> using wireless communication or wired communication. The display <NUM> and the input device <NUM> function as a user interface. The display <NUM> provides various types of information, which is related to the generation of the composite image, to a user. The display <NUM> is not particularly limited, and examples of the display <NUM> include a liquid crystal monitor, a LED monitor, and the like. Further, the input device <NUM> is operated by a user so that various instructions related to the generation of the composite image are input. The input device <NUM> is not particularly limited, and examples of the input device <NUM> include a keyboard, a touch pen, a mouse, and the like. A touch panel display in which the display <NUM> and the input device <NUM> are integrated with each other may be employed.

<FIG> is a functional block diagram showing an example of the configuration that is related to the functions of the image processing device <NUM> according to the present embodiment. As shown in <FIG>, the image processing device <NUM> comprises an image acquisition unit <NUM>, a characteristic parameter acquisition unit <NUM>, a composition unit <NUM>, and an image output unit <NUM>. For example, the processor <NUM> of the image processing device <NUM> according to the present embodiment executes the image processing program <NUM> stored in the storage unit <NUM> to function as the image acquisition unit <NUM>, the characteristic parameter acquisition unit <NUM>, the composition unit <NUM>, and the image output unit <NUM>.

The image acquisition unit <NUM> has a function of acquiring the image data of the projection image, which is projected from the OLED <NUM>, from the smartphone <NUM>. For example, the image processing device <NUM> according to the present embodiment acquires the image data of the projection image, the image data of a captured image captured by the camera <NUM> during the projection of the projection image, and position data from the smartphone <NUM> in association with each other at a predetermined timing, and stores the image data of the projection image, the image data of the captured image, and the position data, which are acquired, in the storage unit <NUM>. For this reason, the image acquisition unit <NUM> acquires the image data of the projection image and the image data of the captured image from the smartphone <NUM> via the I/F unit <NUM> at an arbitrary timing. Further, in a case where a composite image is to be generated, the image acquisition unit <NUM> acquires the image data of the projection image, the image data of the captured image, and the position data from the storage unit <NUM>. In a case where a composite image is to be generated, the image acquisition unit <NUM> outputs the image data of the projection image, the image data of the captured image, and the position data, which are acquired, to the composition unit <NUM>.

The characteristic parameter acquisition unit <NUM> has a function of acquiring parameters (hereinafter, referred to as characteristic parameters) representing the characteristics of the glasses-type information display device <NUM> from the smartphone <NUM>. In the present embodiment, as shown in <FIG>, the characteristic parameter acquisition unit <NUM> acquires, as an example of the characteristic parameters, a brightness a1 (cd/m<NUM>) of the projection image projected from the OLED <NUM>, a brightness a2 (cd/m<NUM>) that is preset as the brightness of the projection image perceived by a user, a brightness b1 (cd/m<NUM>) of the captured image captured by the camera <NUM>, and a brightness b2 (cd/m<NUM>) that is preset as the brightness of the projection image visually recognized by a human through the light guide plate <NUM>. The characteristic parameter acquisition unit <NUM> outputs the acquired characteristic parameters to the composition unit <NUM>.

Specific values of the brightnesses a1, a2, b1, and b2 may be derived from design on the basis of the transmittance or the reflectance of each member, or values obtained from the measurement of a prototype may be set as the specific values, that is, values obtained from the measurement of the AR glasses <NUM> described here or the like may be set as the specific values.

Further, a method of acquiring the characteristic parameters by the characteristic parameter acquisition unit <NUM> is not particularly limited. For example, in a case where the smartphone <NUM> of the glasses-type information display device <NUM> stores the characteristic parameters (the brightnesses a1, a2, b1, and b2) and the characteristic parameters are associated with the image data of the projection image, the characteristic parameter acquisition unit <NUM> may acquire the characteristic parameters associated with the projection image acquired by the image acquisition unit <NUM>. Furthermore, for example, in a case where information indicating the model of the AR glasses <NUM> is associated with the image data of the projection image, the characteristic parameter acquisition unit <NUM> may acquire the information indicating the model of the AR glasses <NUM> associated with the image data of the projection image and may acquire characteristic parameters corresponding to the acquired information indicating the model from an external device or the like through, for example, the network <NUM>.

The composition unit <NUM> has a function of generating a composite image by performing image processing on dimensional data of the brightness of at least one of the projection image or the captured image and superimposing the projection image on the captured image. For example, the composition unit <NUM> of the present embodiment has a function of generating a composite image in which the projection image is combined with the captured image on the basis of the characteristic parameters. The composite image is an image in which a visually-recognized image in which the projection image is inserted into the visual field of a real image of a user who uses the AR glasses <NUM> is reproduced. That is, the composite image generated by the composition unit <NUM> of the present embodiment is an image that simulates the visibility of the projection image of the visually-recognized image, in other words, the transmission state of a virtual image.

The visibility of the projection image, which is the transmission state of the virtual image, will be described here.

<FIG> shows an example of a visually-recognized image <NUM> that is visually recognized by a user using the AR glasses <NUM>. Since the AR glasses <NUM> are a transmission-type display, a virtual image <NUM> visually recognized via the light guide plate <NUM> in a case where the projection image is projected from the OLED <NUM> is visually recognized in a state where a real image <NUM> is transmitted.

In this state, in the related art, as shown in <FIG>, image data of a projection image <NUM> projected from the OLED <NUM> are combined with image data of a captured image <NUM> captured by the camera <NUM> as they are, so that image data of a composite image are generated. In the composite image <NUM> in the related art generated as in <FIG>, the appearance of the projection image <NUM> with respect to the captured image <NUM> may be different from the appearance of the virtual image <NUM> with respect to the real image <NUM> as shown in <FIG>. Particularly, in a case where the image data of the projection image <NUM> are combined with the image data of the captured image <NUM> as they are as shown in <FIG> as in a case where the images are displayed on a non-transmission-type display, such as the display 16A of the smartphone <NUM>, a case where the images are printed by the printer <NUM>, or the like, the composite image <NUM> is generated. In this case, the captured image <NUM> superimposed on the projection image <NUM> is not seen through the projection image <NUM>. Further, even though the image is displayed on a transmission-type display, in a case where the model of the transmission-type display of the AR glasses <NUM> and the model of the transmission-type display on which the projection image <NUM> and the captured image <NUM> are displayed are different from each other and parameters representing the characteristics of the models are different from each other, the transmission state of the projection image <NUM> may be different from the transmission state of the virtual image <NUM>.

Accordingly, the image processing device <NUM> according to the present embodiment generates a composite image which simulates the transmission state of the virtual image <NUM> via the AR glasses <NUM> and in which the captured image <NUM> and the projection image <NUM> are combined with each other.

As shown in <FIG>, the composition unit <NUM> of the present embodiment thins out the image data of the projection image on the basis of the characteristic parameters to generate a thinned image that simulates the transmission state of a virtual image visually recognized by a user using the AR glasses <NUM>. Specifically, the composition unit <NUM> thins out the image data of the projection image on the basis of a brightness ratio a/b unique to the model of the glasses-type information display device <NUM>. Here, a denotes a ratio of the brightness a2 of the projection image, which is perceived by a user, to the brightness a1 of the projection image output from the OLED <NUM>, and is represented by the following equation (<NUM>). Further, b denotes a ratio of the brightness b2 of a real image, which is perceived by a user, to the brightness b1 of the captured image captured by the camera <NUM>, and is represented by the following equation (<NUM>). Each of a and b may have a spatial distribution in a display surface of the AR glasses <NUM>. <MAT> <MAT>.

Furthermore, as shown in <FIG>, the composition unit <NUM> registrates and combines the image data of the thinned image and the image data of the captured image on the basis of the position data to generate the image data of a composite image.

<FIG> shows an example of the composite image <NUM> that is combined by the composition unit <NUM>. Since a thinned image <NUM> is superimposed on the captured image <NUM> as shown in <FIG>, the appearance of the thinned image <NUM> with respect to the captured image <NUM> is the same as the appearance (see <FIG>) of the virtual image <NUM> with respect to the real image <NUM>.

The image output unit <NUM> has a function of outputting image data, which represent the composite image generated by the composition unit <NUM>, to an arbitrary device. The image output unit <NUM> of the present embodiment outputs the image data of the composite image to at least one of the smartphone <NUM> or the printer <NUM>.

Next, the action of the image processing device <NUM> according to the present embodiment will be described. <FIG> is a flowchart showing an example of the flow of image processing that is performed by the processor <NUM> of the image processing device <NUM> according to the present embodiment. For example, in the image processing device <NUM> according to the present embodiment, the processor <NUM> executes the image processing program <NUM> stored in the storage unit <NUM> to perform the image processing of which an example is shown in <FIG> in a case where an instruction to display the composite image is received from the smartphone <NUM> or the printer <NUM>.

In Step S100 of <FIG>, the image acquisition unit <NUM> acquires the image data of the projection image, the image data of the captured image, and the position data, which are associated with each other, from the storage unit <NUM> as described above.

In the next step S102, the characteristic parameter acquisition unit <NUM> acquires the characteristic parameters of the glasses-type information display device <NUM> that has projected the projection image acquired in Step S100 as described above.

In the next step S104, the composition unit <NUM> derives a brightness ratio a/b, which is unique to the model of the glasses-type information display device <NUM>, using Equation (<NUM>) and Equation (<NUM>) as described above.

In the next step S106, the composition unit <NUM> thins out the projection image on the basis of a/b derived in Step S104 to generate a thinned image. Specifically, the composition unit <NUM> generates a thinned image in which a brightness value is set to <NUM> at a ratio of one pixel for every N pixels on the basis of a/b. N is <NUM> or more and is predetermined according to a/b. Unlike in the present embodiment, a brightness value may be reduced from an original value instead of setting a brightness value to <NUM>. In this case, for example, the degree of a reduction in a brightness value may be predetermined according to a/b. It is preferable that processing, such as gradation transformation, is performed on the thinned image.

In the next step S108, the composition unit <NUM> registrates and combines the thinned image generated in Step S106 and the captured image acquired in Step S100 on the basis of the position data to generate a composite image. Specifically, the composition unit <NUM> combines the thinned image and the captured image on the basis of the position data while a spatial position of a subject in a real space is used as a reference. A method of combining the projection image with the captured image is not particularly limited, and for example, an image composition (alpha blending) technique using an alpha channel, or the like may be applied. Examples of a technique related to alpha blending include a technique disclosed in <CIT>.

In the next step S110, the image output unit <NUM> outputs the image data of the composite image, which is generated in Step S108, to the smartphone <NUM> or the printer <NUM> to which an instruction to display the composite image is given. In a case where the processing of Step S110 ends, the image processing shown in <FIG> ends.

A method of generating a composite image by the composition unit <NUM> is not limited to the above-mentioned embodiment. An aspect in which the composition unit <NUM> generates a thinned image on the basis of a/b has been described in the embodiment, but the present disclosure is not limited to this aspect. For example, as shown in <FIG>, the composition unit <NUM> may registrate the image data of the projection image and the image data of the captured image on the basis of the position data and may combine the image data of the projection image with the image data of the captured image by alpha blending using an alpha value corresponding to a/b to generate the image data of the composite image. In this case, a correspondence relationship between a value of a/b and the alpha value may be predetermined. Generally, in a case where the alpha value is in the range of <NUM> to <NUM>, a case where the alpha value is <NUM> indicates a completely transparent state and a case where the alpha value is <NUM> indicates a completely opaque state.

An aspect in which a composite image simulating a visually-recognized image actually visually recognized by a user is generated as an image simulating the transmission state of the virtual image has been described in the above-mentioned embodiment, but an image to be generated is not particularly limited as long as the transmission state of the virtual image is simulated. For example, in a case where a user visually recognizes a specific real world using the AR glasses <NUM>, the image data of a captured image of the real world and the image data of a projection image may be combined with each other as described above to generate the image data of a composite image, which simulates the transmission state of a virtual image, in order to simulate what a visually-recognized image looks like. Further, as shown in <FIG>, the image data of a CG image generated using computer graphics (CG) and the image data of a projection image may be combined with each other as described above to generate the image data of a composite image simulating the transmission state of a virtual image. In this case, the registration of the image data of the projection image and the image data of the CG image may be appropriately adjusted by an operator who generates a composite image. Furthermore, a/b, which is a characteristic parameter, may also be appropriately set depending on AR glasses <NUM> to be assumed. The image data of the projection image may also be an image that is tentatively generated using CG.

As described above, according to this modification example, the composite image is not limited to a visually-recognized image that is actually visually recognized by a user, and a composite image, which indicates a visually-recognized image visually recognized by a user in a case where the AR glasses <NUM> are used, can be generated.

The projection image may include a plurality of frames and the plurality of respective frames may be displayed at intervals, or may be, for example, a video. In this case, the composition unit <NUM> determines a display time of one frame of the projection image on the basis of a/b as shown in <FIG>. The image data of a captured image and the image data of the projection image are combined with each other according to the display time to generate the composite image data of a composite image. A correspondence relationship between the display time and a/b may be predetermined.

As described above, the image processing device <NUM> according to the present embodiment performs composition processing for generating an image, which simulates the transmission state of a virtual image via the glasses-type information display device <NUM>, on the basis of parameters representing the characteristics of the model of the glasses-type information display device <NUM> from a projection image that serves as the basis of the virtual image and a captured image that indicates a real image.

As described above, according to the image processing device <NUM> of the present embodiment, since the characteristic parameters of the model of the glasses-type information display device <NUM> are used, the transmission state of a virtual image can be appropriately simulated. Therefore, according to the image processing device <NUM> of the present embodiment, a visually recognized state shown on a transmission-type display can be accurately reproduced.

An aspect in which the composition unit <NUM> generates a composite image on the basis of a/b as the characteristic parameter has been described in the embodiment, but another parameter representing the characteristics of a model may also be further used in addition to a/b as the characteristic parameters. Examples of the other parameter include at least one of aberration, flare, stray light, or scattering. Which parameter is to be used can be appropriately determined according to the simulation accuracy of the transmission state of a virtual image in a composite image, the characteristics of the glasses-type information display device <NUM>, and the like.

An aspect in which a projection image is visually recognized with the right eye of a user has been described in the embodiment, but the present disclosure is not limited to this aspect. A projection image may be visually recognized with the left eye of a user. In this case, the transmission unit <NUM> for a left eye is an example of the transmission unit of the present disclosure.

The captured image may be a static image or may be a video.

Further, the composition unit <NUM> is not limited to an aspect in which the composition unit <NUM> uses the detection result (position data) of the space recognition sensor <NUM> for registration in a case where a projection image and a captured image are to be combined with each other. For example, a projection image may be superimposed at a predetermined position on a captured image.

Further, an aspect in which the image processing device <NUM> comprises the image acquisition unit <NUM>, the characteristic parameter acquisition unit <NUM>, the composition unit <NUM>, and the image output unit <NUM> has been described in each embodiment, but at least one of the smartphone <NUM>, the smartphone <NUM>, or the printer <NUM> may comprise some or all of these functional units.

Furthermore, the following various processors can be used in the embodiment as the hardware structures of processing units, such as the image acquisition unit <NUM>, the characteristic parameter acquisition unit <NUM>, the composition unit <NUM>, and the image output unit <NUM>, which perform various types of processing. The various processors include a programmable logic device (PLD) that is a processor of which the circuit configuration can be changed after manufacture, such as a field programmable gate array (FPGA), a dedicated electrical circuit that is a processor having circuit configuration dedicatedly designed to perform specific processing, such as an application specific integrated circuit (ASIC), and the like in addition to a CPU that is a general-purpose processor functioning as various processing units by executing software (program) as described above.

One processing unit may be formed of one of these various processors, or may be formed of a combination of two or more processors of the same type or different types (for example, a combination of a plurality of FPGAs or a combination of a CPU and an FPGA). Further, a plurality of processing units may be formed of one processor.

As an example where a plurality of processing units are formed of one processor, first, there is an aspect in which one processor is formed of a combination of one or more CPUs and software as typified by a computer, such as a client or a server, and functions as a plurality of processing units. Second, there is an aspect where a processor fulfilling the functions of the entire system, which includes a plurality of processing units, by one integrated circuit (IC) chip as typified by System On Chip (SoC) or the like is used. In this way, various processing units are formed using one or more of the above-mentioned various processors as hardware structures.

In addition, more specifically, electrical circuitry where circuit elements, such as semiconductor elements, are combined can be used as the hardware structures of these various processors.

Claim 1:
An image display system comprising:
an information display device (<NUM>), wherein the information display device comprises a transmission type display device (<NUM>) displaying to a user a real image and a projection image superimposed on the real image;
a camera (<NUM>) to capture an image of the real image;
a non-transmission type display (<NUM>,<NUM>); and an image processing device (<NUM>), wherein the image processing device comprises an image acquisition unit (<NUM>), a characteristic parameter acquisition unit (<NUM>), a composition unit (<NUM>), and an image output unit (<NUM>);
wherein the composition unit (<NUM>) is configured to perform composition processing for generating a composite image, which simulates a transmission state of the projection image (<NUM>) via the transmission-type display, for display on the non-transmission type display, on the basis of a parameter representing characteristics of a model of the transmission-type display from the projection image (<NUM>) and the real image (<NUM>), as captured by the camera; and
wherein the characteristic parameter acquisition unit (<NUM>), is configured to acquire characteristic parameters of the model of transmission type display device, the characteristics comprising;
a brightness (a1) of the projection image, a brightness (a2) that is preset as the brightness of the real image being perceived by a user, a brightness (b1) of the image captured by the camera (<NUM>), and a brightness (b2) that is preset as the brightness of the projection image seen by the user.