Patent Publication Number: US-2006007245-A1

Title: Image composing system and a method thereof

Description:
CROSS REFERENCE TO RELATED APPLICATION(S)  
      This application is a divisional of U.S. application Ser. No. 10/291,854, filed Nov. 8, 2002, which claims priority of Japanese patent application number 2002-126952, filed on Apr. 26, 2002, priority of which are claimed herein. 
    
    
     TECHNICAL FIELD OF THE INVENTION  
      The present invention relates to an image composing apparatus and method, and more particularly an image composing apparatus and method that can compose images at high speed and in which the apparatus configuration can easily be expanded in proportion to the number of images to be composed.  
     PRIOR ART  
      To intuitively grasp the results of computer processing, the results must be presented in visible form. However, as the scale of processing becomes larger, it becomes difficult to generate necessary images using a single computer. Therefore, it is often practiced to compose images using a plurality of computers.  
       FIG. 1  is a block diagram of an image composing apparatus proposed in the prior art, which comprises N node computers F 1 , F 2  . . . F n  and an N×N crossbar switch S for enabling the N node computers F 1 , F 2  . . . F N  to communicate with each other.  
      Each of image composing boards C 1 , C 2  . . . C N  which are dedicated cards for composing images, is connected to a PCI bus on each node computer F n , and each of the image composing boards C 1 , C 2  . . . C N  are connected to each other via the crossbar switch S.  
      In the image composing apparatus having the above configuration, the image composing board C n  on one node computer F n  composites an image generated in itself with an image generated in the image composing board C m  (m≈n) on another node computer F m , and the composed image is transferred to the third node computer F k  via the crossbar switch S. This process is repeated till a final image is generated.  
      The above image composing apparatus uses a method in which the images generated by separate node computers are composed together based on a parameter called “Z value” that denotes an image composing order. Though the apparatus can respond flexibly to increasing the number of node computers, it involves following problems.  
      (1) The Z value denoting the depth from a viewpoint must be added for data of each pixel, but since the Z value must be highly accurate compared with other information (red component R, green component G, blue component B, and transparency α), adding the Z value almost doubles the number of bits in the image data, and the amount of image data to be transmitted increases.  
      (2) To avoid collisions during transferring image data, a waiting time is necessary, and this waiting time increases the time required for transferring image data.  
      (3) In order to reduce the image data transferring time, the hardware configuration must be increased, because the crossbar switch, for example, should be specially designed to match with each node computer.  
      Therefore, in the above image composing apparatus, when the number of node computers is increased.  
      (1) The scale of the crossbar switch may exceed the practically feasible scale as it increases as the square of the number of node computers.  
      (2) If the number of node computers is limited to the practically feasible number, the time required for establishing synchronization between the images to be composed increases, and an image composing time increases.  
     SUMMARY OF THE INVENTION  
      The present invention has been devised in view of the above problems and an object of the invention is to provide an image composing apparatus and method that can compose images at high speed and in which the apparatus configuration can be easily expanded as the number of images to be composed increases.  
      According to an image composing apparatus and method of a first invention, N (2≦N) sub-images are merged into a single image based on an occlusion relationship between the N sub-images.  
      According to an image composing apparatus and method of a second invention, the occlusion relationship is expressed in the form of a binary space-partitioning tree.  
      According to an image composing apparatus and method of a third invention, input sub-images are composed to generate an output sub-image by iteratively composing two sub-images into one image based on the occlusion relationship between two sub-images.  
      According to an image composing apparatus and method of a fourth invention, the two sub-image composing process compares priorities which denote the occlusion relationship between the two sub-images, and generates a composed image based on the result of the priority comparison by composing one sub-image which is defined as a first sub-image with another sub-image which is defined a second sub-image.  
      According to an image composing apparatus and method of a fifth invention, the two sub-images are composed together in accordance with the following equation.  
               I   ⁢           ⁢       (     R   ,   G   ,   B     )     M       =       I   ⁢           ⁢       (     R   ,   G   ,   B     )     1     *       I   ⁡     (   α   )       1       +     I   ⁢           ⁢       (     R   ,   G   ,   B     )     2                       I   ⁢           ⁢       (   α   )     M       =     I   ⁢           ⁢       (   α   )     1     *   I   ⁢           ⁢       (   α   )     2                 
 
 Where I(R,G,B) M  are the RGB components of the composed image, I(R,G,B) 1  are the RGB components of the first sub-image, I(R,G,B) 2  are the RGB components of the second sub-image, I(α) M  is the transparency of the merged image, I(α) 1  is the transparency of the first sub-image, and I(α) 2  is the transparency of the second sub-image. 
 
      According to an image composing apparatus and method of a sixth invention, the sub-images are synchronized to each other before merging the sub-images.  
      According to an image composing apparatus and method of a seventh invention, N sub-images are sequentially stored in N FIFO storing means, and when the head of any one of the N sub-images arrives at any one of the N FIFO storing means, an instruction to output the stored sub-images in the order in which the sub-images were stored is issued to the N FIFO storing means.  
      According to an image composing apparatus and method of an eighth invention, the plurality of sub-image data have frame structures which contain a priority information at the top of the frame and a sub-image information at the end of the frame. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       FIG. 1  is a diagram showing the configuration of a conventional image composing apparatus.  
       FIG. 2  is a diagram showing the configuration of a composed image generating system using an image composing apparatus according to the present invention.  
       FIG. 3  is a diagram showing the configuration of the image composing apparatus according to the present invention.  
       FIG. 4  is a timing diagram for data transfer between the image composing apparatus and node computers.  
       FIG. 5  is a sub-image output-timing diagram of the node computers.  
       FIG. 6  is a diagram showing the configuration of a synchronization circuit.  
       FIG. 7  is a diagram showing the configuration of a control signal generating circuit.  
       FIG. 8  is a diagram showing the format of sub-image data.  
       FIG. 9  is a bit allocation diagram for time slots.  
       FIG. 10  is a diagram for explaining a method of partitioning space into subspaces.  
       FIG. 11  is diagram showing a binary space partitioning tree representation.  
       FIG. 12  is a diagram for explaining an image blending process.  
       FIG. 13  is a diagram showing the configuration of an image merging circuit.  
       FIG. 14  is a diagram showing the configuration of a multi-layer image composing apparatus. 
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS  
       FIG. 2  is a diagram showing the configuration of a composed image generating system using an image composing apparatus according to the present invention. As shown, the composed image generating system  2  comprises a server  20  which controls the entire apparatus, a display device (for example, a CRT)  21  which displays an image, a plurality of nodes (in the present embodiment, eight nodes)  231  to  238  which generate sub-image signals for a plurality of subspaces created by dividing the three-dimensional space in which an analysis model is constructed, and the image composing apparatus  24  which composes the outputs of the nodes  231  to  238  and transmits a composed result to the server  20 .  
      The server  20  comprises a CPU  201 , a memory  202  which stores a program to be executed by the CPU  201  and the result of processing performed by the CPU  201 , a graphics board (GB)  203  which outputs an image signal to the display apparatus  21 , and an interface board (IFB)  204  for interfacing with the image composing apparatus  24 .  
      The nodes  231 - 238  have an identical configuration, and each node comprises a CPU  23   b  which performs simulation for the subspace in which a portion of an analysis target is constructed, a memory  23   b  which stores a simulation program to be executed by the CPU  23   b  and the simulation result performed by the CPU  23   b,  an image generating engine board (VGB)  23   c  which generates an image representing the result of the simulation for the subspace, and an interface board (IFB)  23   d  for interfacing with the image composing apparatus  24 .  
       FIG. 3  is a diagram showing the configuration of the image composing apparatus, according to the present invention, which comprises a synchronizing section  240 , a first-layer image composing section  241 , a second-layer image composing section  242 , a third-layer image composing section  243 , and an image combining circuit  244 .  
       FIG. 4  is a data transferring timing diagram between the image composing apparatus and the node computers. As shown, the image composing apparatus  24  receives a parameter necessary for image generation from the server and distributes the parameter to the respective node computers  231 - 238  every predetermined period, and each of the node computers  231 - 238  receives the parameter and generates a sub-image.  
      After generating sub-images, the node computers  231 - 238  transfer the sub-images to the image composing apparatus  24 , which composes the sub-images output from the node computers  231 - 238  and transfers the composed image to the display device  21  though the server  20 .  
      Because each of the node computers  231 - 238  generally works independently, synchronization of the timings when the sub-images arrive at the image composing apparatus  24  is not guaranteed.  
       FIG. 5  is a sub-image output timing diagram of the node computers, and this shows a case where the first sub-image of #n frame is output from the first node computer  231  and the last sub-image of #n frame is output from the eighth node computer  238 .  
      In this case, if each of the node computers  231 - 238  stores the entire of #n frame, and begins to transfer the #n frame to the first-layer image composing section  241  after the last part of the #n frame output from the last node computer (in this example, the eighth node computer  238 ) is stored, synchronization among the sub-images can be achieved.  
      However, storing the entire of #n frame requires not only a large capacity of memory, but also long time to store the entire of #n frame.  
      Therefore, the image composing apparatus of the present invention accomplishes not only synchronizing among sub-images, but also shortening the synchronizing time and minimizing the memory capacity, by providing a synchronizing section  240  at the interface of the node computers, and beginning to transfer the frame of the sub-image to the first-layer image composing section  241  at the timing when the beginning parts of the sub-images frame output from node computers are aligned.  
      As shown in  FIG. 3 , the synchronizing section  240  in the image composing apparatus of the present invention comprises a control signal generating circuit A and four synchronization circuits B 1 -B 4  which have identical configurations.  
       FIG. 6  is a diagram showing the configuration of each synchronization circuit, and  FIG. 7  is a diagram showing the configuration of the control signal generating circuit.  
      As shown, the synchronization circuit B i  has two channels: the first channel comprises a FIFO memory  611  and a command decoder  612  connected in parallel with the FIFO memory  611 , while the second channel comprises a FIFO memory  621  and a command decoder  622  connected in parallel with the FIFO memory  621 .  
      The first channel works as follows. The FIFO memory  611  sequentially stores the pixel data of the sub-image frame output from the node computer connected to the first channel until READ READY output from the control signal generating circuit A has been received, and outputs the pixel data (I X ) in the same order when they were stored.  
      The command decoder  612  extracts a priority set (a channel enable and a set priority) contained in the sub-image frame output from the node computer connected to the first channel, and a frame start command and a status sense contained in the frame data. The start command is transferred to the control signal generating circuit A and a 9-bit priority value (P X ) contained in the set priority is transferred to the first-layer image merging section.  
      The operation of the second channel is the same as that of the first channel, except that the pixel data output from the FIFO memory  621  is I Y  and the priority value is P Y  output from the command decoder.  
       FIG. 8  is a diagram showing the format of the sub-image frame output from each node computer, and  FIG. 9  is a bit allocation diagram for time slots. Each of the frame computers  231 - 238  outputs a hardware initialization frame to initialize the image composing apparatus at first, and then sequentially outputs the sub-image frames.  
      The hardware-initializing frame contains a FIFO reset slot and four dummy slots, and resets the FIFO memory in the synchronization circuit.  
      The image frame comprises the priority set and frame data, and the priority set contains a channel enable slot and a set priority slot.  
      The channel enable slot indicates that the channel is in use, and the set priority slot carries the priority given to the sub-image generated by the node computer.  
      The frame data comprises a frame start slot, a status sense slot, a predetermined number of pixel data slots, and a frame end slot.  
      The frame start slot denotes the beginning of the frame data, and the frame end slot denotes the end of the frame data.  
      Each of the pixel data slots stores the red, green, and blue components and the transparency (α value) of each pixel in the sub-image.  
      Each slot consists of 36 bits, which are allocated as shown in  FIG. 9  according to the kind of the slot.  
      For example, in the set priority slot, the priority is set by using the lower 9 bits in No. 0-15 bits, and “4” is set in No. 24-31 bits and “2” in No. 32-35 bits to show that this slot is a set priority slot.  
      Likewise, in the pixel data slot, “1” is set in No. 32-35 bits to show that this slot is a pixel data slot. Further, No. 24-31 bits store the red component, No. 16-23 bits store green component, No. 8-15 bits store the blue component, and No. 0-7 bits store the transparency.  
      The control signal generating circuit A shown in  FIG. 7  comprises eight-input AND gate  7 , and the frame start slots extracted by the command decoders in the respective synchronization circuits are transferred to the inputs of AND gate  7 .  
      Then, when the eight frame start slots of #n frame are detected, a FIFO read signal, that is, the output of the AND gate  7 , becomes “ON”, whereupon the frame data stored in the FIFO memories in the synchronization circuits are read out in the same order when they were stored, and are transferred to the first-layer image composing section.  
      The pixel data are maintained in the FIFO memories until all of the starting portions of the eight #n frames output from the node computers  231 - 238  arrive at the synchronizing section. When the starting portion of the eighth #n frame arrives at the synchronizing section, the control signal generating circuit A sets the FIFO read signal ON, and the data of the eight frames is simultaneously transferred to the first-layer image merging section  241 . The synchronization among the frame data is accomplished.  
      Since a single clock controls the image composing apparatus  24 , synchronization is guaranteed in the processing after the first-layer image merging section  241 .  
      The first-layer image composing section  241  comprises four image composing circuits C 1 -C 4 , the second-layer image merging section  242  comprises two image composing circuits C 5  and C 6 , and the third-layer image composing section  243  comprises one image merging circuit C 7 . The image composing circuits C i  (1≦i≦7) have identical configurations.  
      In the first-layer image composing section  241 , the first image composing circuit C 1  composes the sub-images generated by the first node computer  241  and the second node computer  242  after being synchronized each other. Likewise, the sub-images generated by the third node computer  243  and the fourth node computer  244  are composed in the second image composing circuit C 2 , the sub-images generated by the fifth node computer  245  and the sixth node computer  246  are composed in the third image composing circuit C 3 , and the sub-images generated by the seventh node computer  247  and the eighth node computer  248  are composed in the fourth image composing circuit C 4 .  
      In the second-layer image composing section, the sub-images output from the first image composing circuit C 1  and the second image composing circuit C 2  are composed in the fifth image composing circuit C 5 , while the sub-images output from the third image composing circuit C 3  and the fourth image composing circuit C 4  are composed in the sixth image composing circuit C 6 .  
      In the third-layer image composing section, the sub-images output from the fifth image composing circuit C 5  and the sixth image composing circuit C 6  are composed by the seventh image composing circuit C 7 .  
      Hereafter, the sub-image composing method to be employed in the image composing apparatus of the present invention will be described.  
       FIG. 10  is a diagram for explaining how an image space is partitioned into eight subspaces V 1  to V 8 . The sub-image for the respective subspace V i  is generated by each of the node computers  231  to  238 . The number of subspaces is not specifically limited, but to make effective use of the image merging circuits, it will be advantageous to divide the image space into a plurality of subspaces, in particular, 2 3n  subspaces (where n is a positive integer).  
       FIG. 11  is a binary space partitioning tree representation of the subspaces, showing the arrangement of the subspaces Vi as viewed from the viewpoint.  
      Because the image I 1  corresponding to the subspace V 1  is closest to the viewpoint, priority “1” is assigned, and other priorities are assigned in the order of the images I 3 , I 2 , I 4 , I 6 , I 8 , I 5  and I 7 . The priority can be determined by discriminating the sign of
 
 a   m   X   e   +b   m   Y   e   +c   m   Z   e   +d   m 
 
 with respect to the planes separating the respective subspaces. Here, (X e , Y e , Z e ) are the coordinates of the viewpoint, and (a m , b m , c m , d m ) are the coefficients determined according to the planes separating the respective subspaces. 
 
       FIG. 12  is a diagram for explaining how the sub-images I 1  to I 8  are composed. The eight sub-images I 1  to I 8  can be composed by iteratively composing two sub-images in consideration of the fact that higher priority is given to the sub-images closer to the viewpoint.  
      According to the image composing apparatus of the present invention, each sub-image is given priority instead of the Z value, and image composing circuits for composing two sub-images in consideration for the priority are arranged in a layered structure to obtain high-speed image composition and high expandability.  
       FIG. 13  is a diagram showing the configuration of each image composing circuit C i , which comprises a comparator  131 , a resistor  132 , an interchanger  133 , a composer  134 , and a delay  135 .  
      The comparator  131  compares the two priority values P X  and P Y  transferred from the synchronization circuit or the image merging circuits at the preceding stage, controls the interchanger  133  based on the result of the comparison, and outputs the smaller priority value (higher priority) as P Z  through the resistor  132 .  
      Based on the result of the comparison from the comparator  131 , the interchanger  133  interchanges the two pixel data I X  and I Y  transferred from the synchronization circuit or the image merging circuits at the preceding stage.  
      For example, when the value of the set priority P Y  for the sub-image I Y  is smaller than the value of the set priority P X  for the sub-image I X , that is, when the sub-image I Y  is given higher priority, cross paths are selected, and the sub-images I X  and I Y  are set as follows.  
      I Y →I A    
      I X →I B    
      Conversely, when the value of the set priority P Y  for the sub-image I Y  is larger than the value of the set priority P X  for the sub-image I X , that is, when the sub-image I X  is given higher priority, then straight paths are selected, and the sub-images I X  and I Y  are set as follows.  
      I X →I A    
      I Y →I B    
      The composer  134  generates composed pixel data I Z  by composing the two pixel data I A  and I B  in accordance with the composing equation shown below.  
               I   ⁢           ⁢       (     R   ,   G   ,   B     )     z       =       I   ⁢           ⁢       (     R   ,   G   ,   B     )     A       +         I   ⁡     (   α   )       A     *     +   I     ⁢           ⁢       (     R   ,   G   ,   B     )     B                       I   ⁢           ⁢       (   α   )     z       =     I   ⁢           ⁢       (   α   )     A     *   I   ⁢           ⁢       (   α   )     B                 
 
 where I(R,G,B) Z  is the red, green, and blue component data of the composed pixel data I Z , and I(α) Z  is the transparency of the composed pixel data I Z . 
 
      Finally, the composite pixel data I Z  is generated by storing I(R,G,B) Z  in No. 31-8 bits and I(α) Z  in No. 7-0 bits, and is output.  
      The high four bits of the 36-bit pixel data are set to “1” for denoting that the data is pixel data, and the bits to be processed by the interchanger  133  and the composer  134  are the low 32 bits. Therefore, the high four bits are directly transferred through the delay  135  to the output side of the composer  134  and combined with the output of the composer  134  to reproduce the 36-bit pixel data. The delay time of the delay is set equal to the processing time required in the interchanger  133  and the composer  134 .  
      As described above, the seven image composing circuits are arranged in three layers and the eight sub-images are composed to one image.  
      However, as the pixel data and the priority value are separately processed in each image composing circuit in the image composing circuit in the first-layer-third-layer, the pixel data and the priority value are also separately output from the image composing circuit C 7  in the third-layer image composing section  243 .  
      Therefore, to standardize the configuration of the image composing apparatus of the present invention, the image combing circuit  244  ( FIG. 3 ) arranged at the final stage of the image composing apparatus adds the frame start and the frame sense at the head of the pixel data output from the image merging circuit C 7  and the frame end at the tail of the data to reconstruct the data format shown in  FIG. 8 . Further, the priority set determined based on the priority value output from the image merging circuit C 7  is added in the front of the frame data.  
      By applying the above configuration, the image composing apparatus of the present invention (in the above embodiment, the apparatus for composing eight sub-images into one image) produces the output frame of the same format as the input frame, and it becomes possible to increase the number of images to be composed by arranging the above image composing apparatuses in a layered structure.  
       FIG. 14  is a configuration of a multi-layer sub-image composing apparatus constructed by arranging 8-input and 1-output image composing apparatuses in a layered structure. When the number of sub-images to be input is denoted by N, the number of layers S is given by the following equation.
   S=Ceil (log 2   N ) 
      While the above description has dealt with an image composing apparatus for composing sub-images electrically, a plurality of display devices may be arranged side by side and the sub-images may be composed visually by displaying the outputs of the synchronizing section directly on the respective display apparatuses and thus operating the plurality of display devices as a single display apparatus.  
      According to the image composing apparatus and method of the present invention, as the sub-images are composed based on the occlusion relationship between them, it becomes possible to easily cope with an increase in the number of sub-images to be merged.