Patent Publication Number: US-2004041920-A1

Title: Image forming method, image processing apparatus, and image recording apparatus

Description:
BACKGROUND OF THE INVENTION  
       [0001] This invention relates to an image forming method for applying optimization processing for making the quality of a viewing image formed on an output medium to digital image data which have been obtained by an image-capturing device such as a digital camera, an image processing apparatus, and an image recording apparatus.  
       CONVENTIONAL TECHNOLOGY  
       [0002] Nowadays, digital image data obtained by photographing by means of an image-capturing device are distributed through a storage device such as a CD-R (Compact Disk Recordable), a floppy (registered trade mark) disk, or a memory card, or the Internet, and the way of displaying/printing are diversified; for example, they are displayed on a display monitor such as a CRT (Cathode Ray Tube), a liquid crystal display, and a plasma display and a display device of a small-sized liquid crystal monitor of a cellular phone, or they are printed as a hard copy by an output device such as a digital printer, an ink jet printer, and a thermal printer.  
       [0003] Further, it has generally been put into practice that, when digital image data are displayed and/or printed for the purpose of appreciation, various kinds of image processing represented by gradation adjustment, luminance adjustment, color balance adjustment, and sharpness enhancement are applied in order that a desired image quality may be obtained on a display monitor or a hard copy to be used for appreciation.  
       [0004] To cope with the above-mentioned diversified displaying/printing methods, it has been made an effort to enhance the versatility of digital image data obtained through photographing by means of an image pickup device. For an example of it, there is an attempt to standardize a color space expressed by digital RGB (Red, Green, and Blue) signals into a color space independent of image-capturing device characteristic, and at present, most digital image data adopt “sRGB” as a standardized color space (refer to “Multimedia Systems and Equipment, Color Measurement and Management, Part 2-1: Color Management, Default RGB Color Space-sRGB” IEC 61966-2-1). The color space of this sRGB is defined in correspondence to the color reproduction domain of a standard CRT display monitor.  
       [0005] Generally speaking, digital cameras are equipped with an image sensor (a CCD-type image sensor, hereinafter referred to as a CCD simply) having a charge transferring function by a CCD (Charge Coupled Device) and a photoelectric conversion function provided with a color sensitivity by the use of a combination of color filters having a pane pattern. Digital image data to be outputted by a digital camera are image data such that original electrical signals obtained through photoelectric conversion by means of this CCD are subjected to correction processings for the photoelectric conversion function of the image sensor, file transformation/compression processing into a data format of a specified type which is standardized so as to be able to be read and displayed by the use of an image edition software, etc.  
       [0006] As regards the correction for the photoelectric conversion function of an image sensor, for example, gradation correction, cross talk correction of spectral sensitivity, dark current noise suppression, sharpening, white balance adjustment, chroma adjustment, etc. can be cited. Further, as regards the standardized data format of a specified type, for example, “Baseline Tiff Rev. 6. ORGB Full Color Image” adopted as a non-compressed file of an Exif (Exchangeable image file format) file and a compressed data file format based on a JPEG (Joint Photographic Experts Group) format are known.  
       [0007] An Exif file is one based on sRGB. The correction for the photoelectric conversion function of an image sensor is designed in a way to make the image quality become optimum on a display monitor based on sRGB.  
       [0008] For example, a digital camera of any type, so long as it has a function to write tag information indicating that colors are displayed in a standard color space of a display monitor based on sRGB signals (hereinafter referred to as a monitor profile) as an Exif format, and supplementary information indicating model-dependent data such as the number of pixels, the pixel arrangement, and the number of bits per pixel as meta-data in the file header of digital image data and adopts such a data format, can analyze tag information by an image edition software for displaying digital image data on a display monitor (for example, Photoshop made by Adobe Corp.) to urge an operator to change the monitor profile into sRGB or automatically apply a changing processing; therefore, it has become possible that the difference in the device characteristics between different display devices is reduced, and digital image data obtained by a digital camera photographing are appreciated in a desirable state on a display monitor.  
       [0009] Further, for the supplementary information, in addition to the model-dependent information, it is used a tag (code) indicating, for example, information directly relating to the kind of a camera (model) such as the name of a camera, or the code number, information determining the photographing conditions such as the exposure time, shutter speed, aperture value (f-number), ISO sensitivity, luminance value, object distance range, light source, strobe light emitted or not, object area, white balance, zoom magnification, object structure, photographed scene type, reflection light quantity of the strobe light, and photographing chroma, information relating to the kind of the photographic object, etc. An image edition software and an output device are equipped with a function to read the above-mentioned supplementary information to make the image quality of a hard copy more desirable.  
       PROBLEM TO BE SOLVED BY THE INVENTION  
       [0010] Incidentally, an image displayed on a display device such as a CRT display monitor, or a hard copy image printed by an arbitrary one of various kinds of print device has different color reproduction domain depending on the composition of the fluorescent substance or the coloring material used. For example, the color reproduction domain of a CRT display monitor corresponding to the sRGB standard color space has a broad light green and blue domain, and there is a portion not to be reproduced by a hard copy of a silver halide print or an ink jet print in said color reproduction domain. On the contrary, in the cyan domain of an ink jet print or in the yellow domain of a silver halide print, there is a portion not to be reproduced by a CRT display monitor corresponding to the sRGB standard color space (refer to, for example, “Fine Imaging and Digital photography (in Japanese)” p. 444, edited by the publishing committee of The Society of Photographic Science and Technology of Japan, published by Corona Co., Ltd.). On the other hand, there is a possibility that there exists, among photographic object scenes to become an object of photographing, one that exhibits a color belonging to a domain not to be reproduced in any one of these color reproduction domains.  
       [0011] As described in the above, because the color gamut to be able to be recorded has a limit with respect to a color space (including sRGB) which is optimized on the premise that an image is printed or displayed by means of a specified device, in recording information obtained by an image-capturing device, an adjustment by mapping to compress the color gamut to a color gamut which is able to be recorded becomes necessary. As regards the method of mapping, clipping is the simplest method, in which a point of chromaticity outside the color gamut any point belonging to which is able to be recorded is mapped to the nearest point on the border of the color gamut; however, this method makes the gradation outside the color gamut flat, and produces an image giving an impression of disorder to an observer appreciating the image. For this reason, it is generally adopted at present a non-linear compression method in which points of chromaticity of a domain having a chroma higher than a suitable threshold value are smoothly compressed in accordance with the magnitude of the chroma. As the result of this, points of chromaticity also inside the color gamut any point belonging to which is able to be recorded come to be recorded with their chroma compressed. (The details of the mapping method of a color gamut are described in, for example, “Fine Imaging and Digital Photography (in Japanese)” p. 447, edited by the publishing committee of The Society of Photographic Science and Technology of Japan, published by Corona Co., Ltd.)  
       [0012] However, in the case where a compression operation of color gamut and luminance range as described in the above is carried out, owing to the principle of digital images which are recorded by the use of discrete numerical values, gradation information before compression and information before clipping are lost at that point of time, and the original state cannot be restored. This makes a great restriction for the versatility of high-quality digital image data.  
       [0013] For example, in the case where an image recorded in the standard color space sRGB is printed by a print device, it becomes necessary to make mapping again on the basis of the difference between the standard color space of sRGB and the color reproduction domain of the print device. However, because an image recorded in the standard color space sRGB has lost the gradation information of the domain which was once compressed at the time of recording, its smoothness of gradation becomes worse compared to the case where the information obtained by the image-capturing device is directly mapped to the color reproduction domain of the print device. Further, in the case where, due to an inappropriate gradation compression condition at the time of recording, there are problems such that the picture is whitish, the face is dark, and the fill-in of shadows and the whitening of the highlight areas are remarkable, even though the improvement of the image quality is attempted through the change of the setting of gradation, because the gradation information before compression and the information concerning the flattened portion and the whitened portion have already been lost, only a remarkably insufficient improvement can be made as compared to the case where an image is newly formed from the information obtained by the image-capturing device.  
       [0014] The problems which have been described up to now are resulted from it that the information of a broad color gamut and luminance range obtained by an image-capturing device is recorded with the data compressed to the viewing image referred image data in the state of being optimized with the viewing image supposed. In contrast with this, if an image is recorded from the scene-referred image data whose information on the broad color gamut and luminance range obtained by the image-capturing device is not compressed, the improvident loss of information can be prevented. For the standard color space suitable for recording such scene-referred image data, for example, “RIMM RGB” (Reference Input Medium Metric RGB) and “ERIMM RGB” (Extended Reference Input Medium Metric RGB) are proposed (refer to Journal of Imaging Science and Technology, Vol. 45, pp. 418 to 426, 2001).  
       [0015] However, data that are expressed in such a standard color space are not suitable for displaying directly on a display monitor for appreciation. Digital cameras are generally equipped built-in or connected with a display monitor for the purpose of confirming the angle of view before photographing or the photographed content after photographing. In the case where the photographed image data are recorded as viewing image referred image data like sRGB, there is a merit that the image data can be directly displayed on a display monitor without transformation, but in the case where the photographed image data are recorded as scene-referred image data, a processing to transform the data again into the viewing image referred image data for displaying the data becomes essential. Further, it is well known that a silver halide photographic photosensitive material, a solid state image sensor such as a CCD (Charge Coupled Device), or an image pickup tube such as a photo-multiplier is used for photographing; any image photographed by these means includes a certain amount of noises, which become a cause to lower the image quality. For that reason, when data are transformed again from scene-referred image data to viewing image referred image data, it is necessary to practice image quality improving processings such as a smoothing processing and a sharpening processing at the same time. Such double processings as transformation processing and image quality improving processing described in the above increase the load of processing and electrical power consumption, which makes a cause of the lowering of continuous photographing ability and the limitation of the number of photographing frames in the case of battery photographing.  
       [0016] Any one of the image quality improving processings such as a smoothing processing and a sharpening processing has also a bad effect.  
       [0017] In the following, the bad effect accompanied by a smoothing processing and that accompanied by a sharpening processing will be explained.  
       [0018] Concerning the smoothing processing, a median filter, a compression processing through removing data in the frequency band having larger amount of noises, etc. are known; however, they have the undesirable defect that, if noises are excessively suppressed, an artifact, which visually gives an observer an unnatural impression of disorder, is produced, or the fine structures of the image representing the photographic object information which are essentially not to be suppressed are suppressed together with the noises.  
       [0019] Further, in the case where the information volume of the digital image data to become the object of processing (for example, the number of gray scales, the number of pixels, etc.) is small, the cause of noise generation has not been specified, or the means for the smoothing processing is not suitable, the suppression of artifact becomes more difficult.  
       [0020] On the other hand, for the sharpening processing, an unsharp mask, an amplification processing by enhancing data in frequency bands having a large amount of edge components, etc. are known; however, if sharpness is enhanced, conversely, a problem that the noise component contained in the image is also enhanced to deteriorate the image quality is produced.  
       [0021] As explained in the above, because each of a smoothing processing and a sharpening processing has a bad effect based on each processing, an optimum combination of them should be determined on the basis of the size of hard copies produced by the output device, the distance of observation, etc. with the amount of application of the smoothing processing and sharpening processing (the degree of application) taken into consideration. However, the preparation of a processing condition table and the time required for the reference to the processing condition table are necessary, and estimation of the production amount of artifacts is difficult; therefore, there has been a problem that it is difficult to take a suitable countermeasure for suppressing these.  
       [0022] It is an object of this invention to provide an image forming method wherein, when an optimization processing for transforming the data into the color reproduction space of a display monitor or an output device is applied to image data obtained by an image-capturing device, the amount of production of artifacts is small and an efficient image quality improving processing can be carried out, and an image processing apparatus and an image recording apparatus using the method.  
       SUMMARY OF THE INVENTION  
       [0023] The above-mentioned object of this invention can be solved by any one of the following structures to be described below.  
       [0024] (1) An image forming method in which image processing for forming an optimum viewing image on an output medium is applied to captured-image data outputted from an image-capturing device,  
       [0025] said image processing comprising a scene-referred image data generation process for generating scene-referred image data on the basis of said captured-image data and a viewing image referred image data generation process for generating viewing image referred image data on the basis of said generated scene-referred image data,  
       [0026] said image forming method being characterized by it that a smoothing processing is practiced in said scene-referred image data generation process and a sharpening processing is practiced in said viewing image referred image data generation process.  
       [0027] (2) An image forming method in which image processing for forming an optimum viewing image on an output medium is applied to captured-image data outputted from an image-capturing device,  
       [0028] said image processing comprising a scene-referred image data generation process for generating scene-referred image data on the basis of said captured-image data and a viewing image referred image data generation process for generating viewing image referred image data on the basis of said generated scene-referred image data,  
       [0029] said image forming method being characterized by it that a smoothing processing is practiced in said scene-referred image data generation process and a sharpening processing is practiced after viewing image referred image data have been generated in said viewing image referred image data generation process.  
       [0030] (3) An image forming method in which image processing for forming an optimum viewing image on an output medium is applied to captured-image data outputted from an image-capturing device,  
       [0031] said image processing comprising a scene-referred image data generation process for generating scene-referred image data on the basis of said captured-image data and a viewing image referred image data generation process for generating viewing image referred image data on the basis of said generated scene-referred image data,  
       [0032] said image forming method being characterized by it that a smoothing processing is practiced at first and a sharpening processing is practiced next in said scene-referred image data generation process.  
       [0033] (4) An image forming method in which image processing for forming an optimum viewing image on an output medium is applied to captured-image data outputted from an image-capturing device,  
       [0034] said image processing comprising a scene-referred image data generation process for generating scene-referred image data on the basis of said captured-image data and a viewing image referred image data generation process for generating viewing image referred image data on the basis of said generated scene-referred image data,  
       [0035] said image forming method being characterized by it that a smoothing processing is applied to scene-referred image data generated in said scene-referred image data generation process and a sharpening processing is practiced in said viewing image referred image data generation process.  
       [0036] (5) An image forming method in which image processing for forming an optimum viewing image on an output medium is applied to captured-image data outputted from an image-capturing device,  
       [0037] said image processing comprising a scene-referred image data generation process for generating scene-referred image data on the basis of said captured-image data and a viewing image referred image data generation process for generating viewing image referred image data on the basis of said generated scene-referred image data,  
       [0038] said image forming method being characterized by it that a smoothing processing is applied to scene-referred image data generated in said scene-referred image data generation process and a sharpening processing is applied to viewing image referred image data generated in said viewing image referred image data generation process.  
       [0039] (6) An image processing apparatus which applies image processing for forming an optimum viewing image on an output medium to captured-image data outputted from an image-capturing device comprising  
       [0040] a scene-referred image data generation means for generating scene-referred image data on the basis of said captured-image data, and  
       [0041] a viewing image referred image data generation means for applying an image processing for optimizing said scene-referred image data to generate viewing image referred image data, characterized by  
       [0042] said scene-referred image data generation means comprising a smoothing processing means for applying a smoothing processing to said captured-image data, and  
       [0043] said viewing image referred image data generation means comprising a sharpening processing means for applying a sharpening processing to said scene-referred image data.  
       [0044] (7) An image processing apparatus which applies image processing for forming an optimum viewing image on an output medium to captured-image data outputted from an image-capturing device comprising  
       [0045] a scene-referred image data generation means for generating scene-referred image data on the basis of said captured-image data, and  
       [0046] a viewing image referred image data generation means for applying an image processing for optimizing said scene-referred image data to generate viewing image referred image data, characterized by  
       [0047] said scene-referred image data generation means comprising a smoothing processing means for applying a smoothing processing to said captured-image data, and  
       [0048] said viewing image referred image data generation means comprising a sharpening processing means for applying a sharpening processing to said generated viewing image referred image data.  
       [0049] (8) An image processing apparatus which applies image processing for forming an optimum viewing image on an output medium to captured-image data outputted from an image-capturing device comprising  
       [0050] a scene-referred image data generation means for generating scene-referred image data on the basis of said captured-image data, and  
       [0051] a viewing image referred image data generation means for applying an image processing for optimizing said scene-referred image data to generate viewing image referred image data, characterized by  
       [0052] said viewing image referred image data generation means comprising a smoothing processing means for practicing a smoothing processing and a sharpening processing means for practicing a sharpening processing in the generation process of viewing image referred image data by said viewing image referred image data generation means.  
       [0053] (9) An image processing apparatus which applies image processing for forming an optimum viewing image on an output medium to captured-image data outputted from an image-capturing device comprising  
       [0054] a scene-referred image data generation means for generating scene-referred image data on the basis of said captured-image data, and  
       [0055] a viewing image referred image data generation means for applying an image processing for optimizing said scene-referred image data to generate viewing image referred image data, characterized by  
       [0056] said scene-referred image data generation means comprising a smoothing processing means for applying a smoothing processing to said generated scene-referred image data, and  
       [0057] said viewing image referred image data generation means comprising a sharpening processing means for applying a sharpening processing to said scene-referred image data having been subjected to said smoothing processing.  
       [0058] (10) An image processing apparatus which applies image processing for forming an optimum viewing image on an output medium to captured-image data outputted from an image-capturing device comprising  
       [0059] a scene-referred image data generation means for generating scene-referred image data on the basis of said captured-image data, and  
       [0060] a viewing image referred image data generation means for applying an image processing for optimizing said scene-referred image data to generate viewing image referred image data, characterized by  
       [0061] said scene-referred image data generation means comprising a smoothing processing means for applying a smoothing processing to said generated scene-referred image data, and  
       [0062] said viewing image referred image data generation means comprising a sharpening processing means for applying a sharpening processing to said generated viewing image referred image data.  
       [0063] (11) An image recording apparatus which applies image processing for forming an optimum viewing image on an output medium to captured-image data outputted from an image-capturing device and outputs them comprising  
       [0064] a scene-referred image data generation means for generating scene-referred image data on the basis of said captured-image data, and  
       [0065] a viewing image referred image data generation means for applying an image processing for optimizing said scene-referred image data to generate viewing image referred image data, characterized by  
       [0066] said scene-referred image data generation means comprising a smoothing processing means for applying a smoothing processing to said captured-image data, and  
       [0067] said viewing image referred image data generation means comprising a sharpening processing means for applying a sharpening processing to said scene-referred image data.  
       [0068] (12) An image recording apparatus which applies image processing for forming an optimum viewing image on an output medium to captured-image data outputted from an image-capturing device and outputs them comprising  
       [0069] a scene-referred image data generation means for generating scene-referred image data on the basis of said captured-image data, and  
       [0070] a viewing image referred image data generation means for applying an image processing for optimizing said scene-referred image data to generate viewing image referred image data, characterized by  
       [0071] said scene-referred image data generation means comprising a smoothing processing means for applying a smoothing processing to said captured-image data, and  
       [0072] said viewing image referred image data generation means comprising a sharpening processing means for applying a sharpening processing to said generated viewing image referred image data.  
       [0073] (13) An image recording apparatus which applies image processing for forming an optimum viewing image on an output medium to captured-image data outputted from an image-capturing device and outputs them comprising  
       [0074] a scene-referred image data generation means for generating scene-referred image data on the basis of said captured-image data, and  
       [0075] a viewing image referred image data generation means for applying an image processing for optimizing said scene-referred image data to generate viewing image referred image data, characterized by  
       [0076] said viewing image referred image data generation means comprising a smoothing processing means for practicing a smoothing processing and a sharpening processing means for practicing a sharpening processing in the generation process of viewing image referred image data by said viewing image referred image data generation means.  
       [0077] (14) An image recording apparatus which applies image processing for forming an optimum viewing image on an output medium to captured-image data outputted from an image-capturing device and outputs them comprising  
       [0078] a scene-referred image data generation means for generating scene-referred image data on the basis of said captured-image data, and  
       [0079] a viewing image referred image data generation means for applying an image processing for optimizing said scene-referred image data to generate viewing image referred image data, characterized by  
       [0080] said scene-referred image data generation means comprising a smoothing processing means for applying a smoothing processing to said generated scene-referred image data, and  
       [0081] said viewing image referred image data generation means comprising a sharpening processing means for applying a sharpening processing to said scene-referred image data having been subjected to said smoothing processing.  
       [0082] (15) An image recording apparatus which applies image processing for forming an optimum viewing image on an output medium to captured-image data outputted from an image-capturing device and outputs them comprising  
       [0083] a scene-referred image data generation means for generating scene-referred image data on the basis of said captured-image data, and  
       [0084] a viewing image referred image data generation means for applying an image processing for optimizing said scene-referred image data to generate viewing image referred image data, characterized by  
       [0085] said scene-referred image data generation means comprising a smoothing processing means for applying a smoothing processing to said generated scene-referred image data, and  
       [0086] said viewing image referred image data generation means comprising a sharpening processing means for applying a sharpening processing to said generated viewing image referred image data.  
       [0087] (16) An image forming method as set forth in any one of the structures (1) to (5), characterized by  
       [0088] said smoothing processing being carried out by means of a filter to change mask sizes, mask shapes, and threshold values, on the basis of the noise characteristic of image data.  
       [0089] (17) An image processing apparatus as set forth in any one of the structures (6) to (10), characterized by  
       [0090] said smoothing processing being carried out by means of a filter to change mask sizes, mask shapes, and threshold values, on the basis of the noise characteristic of image data.  
       [0091] (18) An image recording apparatus as set forth in any one of the structures (11) to (15), characterized by  
       [0092] said smoothing processing being carried out by means of a filter to change mask sizes, mask shapes, and threshold values, on the basis of the noise characteristic of image data.  
       [0093] (19) An image forming method as set forth in any one of the structures (1) to (5), and (16) characterized by  
       [0094] the amount of application of said sharpening processing being adjusted in accordance with the kind of said output medium.  
       [0095] (20) An image processing apparatus as set forth in any one of the structures (6) to (10), and (17) characterized by  
       [0096] the amount of application of said sharpening processing being adjusted in accordance with the kind of said output medium.  
       [0097] (21) An image recording apparatus as set forth in any one of the structures (11) to (15), and (18) characterized by  
       [0098] the amount of application of said sharpening processing being adjusted in accordance with the kind of said output medium.  
       [0099] (22) An image forming method as set forth in any one of the structures (1) to (5), (16), and (19) characterized by  
       [0100] the amount of application of said sharpening processing being adjusted in accordance with the size of said output medium.  
       [0101] (23) An image processing apparatus as set forth in any one of the structures (6) to (10), (17), and (20) characterized by  
       [0102] the amount of application of said sharpening processing being adjusted in accordance with the size of said output medium.  
       [0103] (24) An image recording apparatus as set forth in any one of the structures (11) to (15), (18), and (21) characterized by  
       [0104] the amount of application of said sharpening processing being adjusted in accordance with the size of said output medium.  
       [0105] (25) An image forming method as set forth in any one of the structures (1) to (5), (16), (19), and (22) characterized by  
       [0106] the amount of application of said sharpening processing being adjusted in accordance with the size of the main photographic object.  
       [0107] (26) An image processing apparatus as set forth in any one of the structures (6) to (10), (17), (20), and (23) characterized by  
       [0108] the amount of application of said sharpening processing being adjusted in accordance with the size of the main photographic object.  
       [0109] (27) An image recording apparatus as set forth in any one of the structures (11) to (15), (18), (21), and (24) characterized by  
       [0110] the amount of application of said sharpening processing being adjusted in accordance with the size of the main photographic object.  
       [0111] (28) An image forming method as set forth in any one of the structures (1) to (5), (16), (19), (22), and (25) characterized by  
       [0112] the amount of application of said sharpening processing being adjusted in accordance with the photographed scene.  
       [0113] (29) An image processing apparatus as set forth in any one of the structures (6) to (10), (17), (20), (23), and (26) characterized by  
       [0114] the amount of application of said sharpening processing being adjusted in accordance with the photographed scene.  
       [0115] (30) An image recording apparatus as set forth in any one of the structures (11) to (15), (18), (21), (24), and (27) characterized by  
       [0116] the amount of application of said sharpening processing being adjusted in accordance with the photographed scene.  
       [0117] (31) An image forming method as set forth in any one of the structures (1) to (5), (16), (19), (22), (25), and (28) characterized by  
       [0118] the captured-image data outputted from said image-capturing device being scene-referred image data.  
       [0119] (32) An image processing apparatus as set forth in any one of the structures (6) to (10), (17), (20), (23), (26), and (29) characterized by  
       [0120] the captured-image data outputted from said image-capturing device being scene-referred image data.  
       [0121] (33) An image recording apparatus as set forth in any one of the structures (11) to (15), (18), (21), (24), (27), and (30) characterized by  
       [0122] the captured-image data outputted from said image-capturing device being scene-referred image data.  
       [0123] (34) An image forming method as set forth in any one of the structures (1) to (5), (16), (19), (22), (25), and (28) characterized by  
       [0124] the captured-image data outputted from said image-capturing device being scene-referred raw data.  
       [0125] (35) An image processing apparatus as set forth in any one of the structures (6) to (10), (17), (20), (23), (26), and (29) characterized by  
       [0126] the captured-image data outputted from said image-capturing device being scene-referred raw data.  
       [0127] (36) An image recording apparatus as set forth in any one of the structures (11) to (15), (18), (21), (24), (27), and (30) characterized by  
       [0128] the captured-image data outputted from said image-capturing device being scene-referred raw data.  
       [0129] (37) An image forming method as set forth in any one of the structures (1) to (5), (16), (19), (22), (25), and (28) characterized by  
       [0130] the captured-image data outputted from said image-capturing device being viewing image referred image data.  
       [0131] (38) An image processing apparatus as set forth in any one of the structures (6) to (10), (17), (20), (23), (26), and (29) characterized by  
       [0132] the captured-image data outputted from said image-capturing device being viewing image referred image data.  
       [0133] (39) An image recording apparatus as set forth in any one of the structures (11) to (15), (18), (21), (24), (27), and (30) characterized by  
       [0134] the captured-image data outputted from said image-capturing device being viewing image referred image data. 
     
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
     [0135]FIG. 1 is a graph showing the correlation between the observation distance and the length of the longer side of an image;  
     [0136]FIG. 2 is a graph showing the correlation between the contrast ratio and the angle of visual field of the main photographic object;  
     [0137]FIG. 3 is a graph showing the correlation between the degree of sharpness enhancement and the angle of visual field of the main photographic object;  
     [0138]FIG. 4 is a graph showing the correlation between the degree of granularity suppression and the angle of visual field of the main photographic object;  
     [0139]FIG. 5 is a graph showing the correlation between the degree of chroma enhancement and the angle of visual field of the main photographic object;  
     [0140]FIG. 6 is a graph showing the correlation between the gain of the middle frequency component and the angle of visual field of the main photographic object;  
     [0141]FIG. 7 is a graph showing the correlation between the gain of the high frequency component and the angle of visual field of the main photographic object;  
     [0142]FIG. 8 is a drawing showing the file structure of digital image data used in this invention;  
     [0143]FIG. 9 is a drawing showing the file structure of digital image data used in this invention;  
     [0144]FIG. 10 is a drawing showing the file structure of digital image data used in this invention;  
     [0145]FIG. 11 is a drawing showing the file structure of digital image data used in this invention;  
     [0146]FIG. 12 is a drawing showing the file structure of digital image data used in this invention;  
     [0147]FIG. 13 is a drawing showing the file structure of digital image data used in this invention;  
     [0148]FIG. 14 is a block diagram showing the functional structure of an image-capturing device  21  relating to this invention;  
     [0149]FIG. 15 is a flow chart showing a saving processing of scene-referred raw data to be practiced by the control of a controller  11  in FIG. 14;  
     [0150]FIG. 16 is a block diagram showing the functional structure of an image processing apparatus  117  of the embodiment 1 of this invention;  
     [0151]FIG. 17 is a flow chart showing an image data generation processing A to be practiced by the cooperation of the pertinent portions of the image processing apparatus  117  shown in FIG. 16;  
     [0152]FIG. 18 is a perspective view showing the outside structure of an image recording apparatus  201  of this invention;  
     [0153]FIG. 19 is a block diagram showing the internal structure of the image recording apparatus  201  of this invention;  
     [0154]FIG. 20 is a block diagram showing the functional structure of the image processor  270  shown in FIG. 19;  
     [0155]FIG. 21 is a flow chart showing an image data generation processing A to be practiced by the cooperation of the pertinent portions of the image processing apparatus  117  shown in FIG. 20;  
     [0156]FIG. 22 is a block diagram showing the functional structure of an image processing apparatus  117  of the embodiment 2 of this invention;  
     [0157]FIG. 23 is a flow chart showing an image data generation processing B to be practiced by the cooperation of the pertinent portions of the image processing apparatus  117  shown in FIG. 22;  
     [0158]FIG. 24 is a flow chart showing an image data generation processing B to be practiced by the cooperation of the pertinent portions of the image recording apparatus  201  shown in FIG. 20;  
     [0159]FIG. 25 is a block diagram showing the functional structure of an image processing apparatus  117  of the embodiment 3 of this invention;  
     [0160]FIG. 26 is a flow chart showing an image data generation processing C to be practiced by the cooperation of the pertinent portions of the image processing apparatus  117  shown in FIG. 25;  
     [0161]FIG. 27 is a flow chart showing an image data generation processing C to be practiced by the cooperation of the pertinent portions of the image recording apparatus  201  shown in FIG. 20;  
     [0162]FIG. 28 is a block diagram showing the functional structure of an image processing apparatus  117  of the embodiment 4 of this invention;  
     [0163]FIG. 29 is a flow chart showing an image data generation processing D to be practiced by the cooperation of the pertinent portions of the image processing apparatus  117  shown in FIG. 28;  
     [0164]FIG. 30 is a flow chart showing an image data generation processing D to be practiced by the cooperation of the pertinent portions of the image recording apparatus  201  shown in FIG. 20;  
     [0165]FIG. 31 is a block diagram showing the functional structure of an image processing apparatus  117  of the embodiment 5 of this invention;  
     [0166]FIG. 32 is a flow chart showing an image data generation processing E to be practiced by the cooperation of the pertinent portions of the image processing apparatus  117  shown in FIG. 31; and  
     [0167]FIG. 33 is a flow chart showing an image data generation processing E to be practiced by the cooperation of the pertinent portions of the image recording apparatus  201  shown in FIG. 20.  
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT  
     [0168] The term “generation” used in the description of this application means that a program or a processing circuit operating in an image-capturing device, an image processing apparatus, or an image recording apparatus of this invention newly generates image signals or data. The term “production” is sometimes used as the synonym.  
     [0169] Further, an “output medium” is a display device such as a CRT, a liquid crystal display, a plasma display, or the like, or a paper material for the production of a hard copy image such as a silver halide photographic paper, an ink jet paper, a thermal printer paper, or the like.  
     [0170] Further, the term “viewing image referred image data” means digital image data to be used for a display device such as a CRT, a liquid crystal display, a plasma display, or the like, or to be used by an output device generating a hard copy image on a paper material for the production of a hard copy image such as a silver halide photographic paper, an ink jet paper, a thermal printer paper, or the like, and means image data having been subjected to an “optimization processing” for obtaining an optimum image on an output medium.  
     [0171] Further, an “image processing for optimization” is a processing for obtaining an optimum image on an output medium such as a CRT, a liquid crystal display, a plasma display, or the like, or a paper material for the production of a hard copy image such as a silver halide photographic paper, an ink jet paper, a thermal printer paper, or the like, and on the premise that an image is displayed, for example, on a CRT display monitor based on the sRGB standard, the image data are processed in such a way as to make an optimum color reproduction be obtained in the color gamut of the sRGB standard. On the premise that an image is outputted on a silver halide photographic paper, the image is processed in a way such that an optimum color reproduction can be obtained in the color gamut of the silver halide photographic paper. Further, in addition to the above-mentioned compression of color gamut, a gradation compression from 16 bits to 8 bits, reduction of the number of pixels outputted, a processing to cope with the output characteristic (LUT) of the output device, etc. are also included. Besides, needless to say, image processing such as noise suppression, sharpening, color balance adjustment, chroma adjustment, or dodging processing is carried out.  
     [0172] Further, the term “an image-capturing device” used in the description of this application means an apparatus equipped with an image sensor having a photoelectric conversion function, and includes what is called a digital camera or a scanner. For an example of the above-mentioned image sensor, an image sensor of a CCD type composed of a CCD (Charge Coupled Device) and a photoelectric conversion part given a color sensitivity with a color filter of pane pattern, and an image sensor of a CMOS type can be cited. The output electric current of these image sensors is digitized by an A/D converter. The content of each color channel is signal strength based on the spectral sensitivity specific to the image sensor.  
     [0173] As regards the kind of the “image-capturing device” to be used in practicing this invention, there are three types to be described below, and different “captured-image data” are outputted from them respectively. The first one, as described in the structures (31) to (33), is of a type to generate “scene-referred image data” through image pickup, the second one, as described in the structures (34) to (36), is of a type to output “scene-referred raw data” depending on the characteristics of the image-capturing device to be used in the generation of “scene-referred image data” and “reproduction assisting data in practicing a correction processing for image-capturing device characteristics”, and the third one, as described in the structures (37) to (39), is of a type to generate “viewing image referred image data”.  
     [0174] The term “scene-referred image data” means image data in the state in which at least the signal strengths of the respective color channels based on the spectral sensitivity of the image sensor itself have already been mapped to a standard color space such as RIMM RGB or ERIMM RGB described in the explanation of the conventional technology, while image processings to alter the data content for improving the effect in the image appreciation such as a gradation transformation, a sharpness enhancement, and a chroma enhancement are omitted. Further, it is desirable that scene-referred image data are data having been subjected to a correction for the photoelectric conversion characteristic of the image-capturing device (the opto-electronic conversion function defined by ISO1452, refer to, for example, “Fine Imaging and Digital Photography (in Japanese) p. 449, edited by the publishing committee of The Society of Photographic Science and Technology of Japan, published by Corona Co., Ltd.).  
     [0175] “Scene-referred raw data depending on the characteristics of the image-capturing device” are crude output data directly outputted from the image-capturing device, are information true to the photographic object, and mean the data as they are digitized by the above-mentioned A/D converter, or data obtained by the application of a correction for a fixed pattern noise and/or a dark current noise etc. to said data; RAW data are included in these. For the above-mentioned scene-referred raw data, image processings for improving the effect at the time of appreciating an image such as a gradation transformation, a sharpness enhancement, and a chroma enhancement, and a processing to map signal strength data of each color channel based on the spectral sensitivity specific to the image sensor into the above-mentioned standardized color space such as RIMM RGB, sRGB, or the like are omitted.  
     [0176] The term “reproduction assisting data in practicing a correction processing for image-capturing device characteristics” means data to make it possible to practice a correction processing for the characteristics of the image-capturing device defined in the preceding paragraph by the utilization of information only noted in the data. It is necessary that, at least, the information to make it possible to map the signal strength data of each color channel based on the spectral sensitivity of the image sensor itself into the above-mentioned standard color space such as RIMM RGB, sRGB, or the like, that is, the matrix coefficients to be used in transforming the signal strength data of each color channel based on the spectral sensitivity of the image sensor into data in the specified standard color space such as RIMM RGB are noted.  
     [0177] Further, for the “scene-referred image data”, in addition to the image data obtained by a digital camera, which is an image-capturing device, image data produced by the use of a film scanner for inputting frame image information recorded on a photographic photosensitive material such as a color negative film, a color reversal film, a black-and-white negative film, and a black-and-white reversal film by means of an analog camera, or image data produced by the use of a flat bed scanner for inputting image information reproduced on a color photographic paper, which is a silver halide photographic paper can be used.  
     [0178] As the input data to be used for an image forming method, an image processing apparatus, and an image recording apparatus, “scene-referred image data” obtained by the use of an “image-capturing device” for generating the above-mentioned “scene-referred image data” are desirable. However, from the viewpoint of reducing the load of processing and the power consumption in the image-capturing device, “scene-referred raw data” depending on the characteristics of the image-capturing device to be used for the generation of the above-mentioned “scene-referred image data”, and “scene-referred raw data” to be obtained by the use of “an image-capturing device” of a type to output “reproduction assisting data in practicing a correction processing for image-capturing device characteristics” are more desirable. Further, in the case where an “image-capturing device of a type to generate the above-mentioned “viewing image referred image data” is used, it is required a processing to transform “viewing image referred image data” into “scene-referred image data” in the image processing apparatus or in the image recording apparatus. To state it concretely, it is a processing to reproduce a state where no difference for every image-capturing device is present through the correction of at least one of “gradation characteristic”, “color reproduction characteristic”, and “frequency characteristic” based on the characteristics of each image-capturing device.  
     [0179] In an image processing apparatus and an image recording apparatus, digital image data to be obtained by an image-capturing device can be obtained by a reading means reading them as stored in any one of portable “media” known to the public such as a compact flash (registered trade mark), a memory stick, a smart medium, a multi-media card, a floppy (registered trade mark) disk, an opto-magnetic recording medium (MO), and a CD-R. In another way, digital image data can be also obtained from a remote site through a communication means such as a network.  
     [0180] Further, as regards a “smoothing processing”, for example, image data are divided into plural frequency ranges by an orthogonal transformation such as a DCT (Discrete Cosine Transformation), a Fourier transformation, or a multi-resolution transformation such as an orthogonal wavelet transformation, and the smoothing of an image is carried out by the use of a low pass filter of the frequency ranges. To state it concretely, the smoothing of an image is carried out by it that, in the basic image by an orthogonal transformation, the component within the radius R with respect to the origin is completely preserved and other components are removed, or by the application of a coring processing to make the component lower than a specified threshold value, for example, zero to image data of a specified resolution obtained by a multi-resolution transformation.  
     [0181] The term “a filter to change mask sizes, mask shapes, and threshold values on the basis of the noise characteristic of image data” described in the structures (16), (17), and (18) means a smoothing filter characterized by comprising a step to specify the surrounding pixels of a pixel of remark to be used for the processing on the basis of the noise of the image input medium, and a step to determine the pixel value of said pixel of remark on the basis of the pixel values of the specified surrounding pixels. For example, by the changing of the mask sizes in accordance with the noise characteristic, or by the changing of the threshold values for the comparison with the difference of the pixel value between a pixel of remark and its surrounding pixels, the range of surrounding pixels to be used in the processing is altered; owing to that, it becomes possible to prevent the lowering of the image quality and make the speed of processing higher.  
     [0182] Further, as regards a smoothing processing of this invention, it is possible to use a method in which digital image data are separated into luminance information and color information, a smoothing processing is applied to the color information only. By this method, without being accompanied by the lowering of sharpness, color unevenness of an image can be reduced.  
     [0183] A “sharpening processing” is a processing for restoring an image deteriorated owing to blur, or enhancing the edges in an image to make it easy to observe in accordance with the purpose. As regards the concrete method of the sharpening processing, enhancement by first order differential operation, enhancement by second order differential operation (Laplacian), etc. can be cited; among them, an enhancement processing by Laplacian and a processing by means of an unsharp mask are widely used, because a comparatively natural enhancement processing can be carried out.  
     [0184] In a processing by Laplacian, enhancement of edges can be made by subtracting the Laplacian at each pixel of the image data of an original image from each pixel value of the image data of the image. With each pixel value in the image data of an original image, that is, an original signal denoted by f(i, j) (i and j represent the coordinates), and each pixel value after being subjected to a processing by Laplacian, that is, a processed signal denoted by F(i, j), the above-mentioned processing is expressed by the following equation:  
       F ( i,j )= f ( i,j )−∇ 2   f ( i,j )  (1).  
     [0185] The expression “the amount of application of said sharpening processing being adjusted in accordance with the kind of said output medium” described in the structures (19), (20), and (21) means that the amount of the above-mentioned edge enhancement, for example, by means of an unsharp mask is adjusted for each kind of the output medium, that is, a display device such as a CRT, a liquid crystal display, or a plasma display, or a paper material for the production of a hard copy image such as a silver halide photographic paper, an ink jet paper, or a thermal printer paper.  
     [0186] Depending on the characteristics of the output system including the output device, not to mention the output medium, sharpness characteristic becomes different even for the output of the same image data; therefore, it can be said that this invention makes it possible to obtain a subjectively desirable image quality.  
     [0187] The expression “the amount of application of said sharpening processing being adjusted in accordance with the size of said output medium” described in the structures (22), (23), and (24) means that the amount of the above-mentioned edge enhancement, for example, by means of an unsharp mask is adjusted for the size of the output medium, more desirably for the size of each kind of the output medium, that is, a display device such as a CRT, a liquid crystal display, or a plasma display, or a paper material for the production of a hard copy image such as a silver halide photographic paper, an ink jet paper, or a thermal printer paper. Depending on the characteristic of the output system including the output device, not to mention the output medium, sharpness characteristic becomes different even for the output of the same image data in accordance with the output size (the same relation can be established for the observation distance between the output medium and the observer); therefore, it can be said that this invention makes it possible to obtain a subjectively more desirable image quality.  
     [0188] The expression “the amount of application of said sharpening processing being adjusted in accordance with the size of the main photographic object” means that the amount of edge enhancement, for example, by means of an unsharp mask is adjusted in accordance with the size of the main photographic object, for example, the face of the person.  
     [0189] The above-mentioned “being adjusted in accordance with the size of the main photographic object” means that the content of the image processing, in particular, the amount of application of sharpening processing to be applied to the image data representing the photographed image is determined so as to give a better impression to a person observing the image on the output medium, on the basis of the “size of the output medium” and the “size of the main photographic object in the viewing image” together with the “size of the viewing image”.  
     [0190] The above-mentioned “main photographic object” means a photographic object which is remarked from its major role in the expression considered, and in most cases, it is a person, in particular, the region including the face. Also in the case of a landscape photograph, it rarely occurs that the composition is set vaguely; in one case, a flower, an animal, or a still-standing object at a short distance is the main photographic object, and in another case, a mountain, the sky, or clouds in the background is the main photographic object.  
     [0191] The above-mentioned “size of the viewing image” means the actual size of the total area of the photographed image when it is printed or displayed on an output medium; for example, in the case it is printed on a paper sheet of A4 size, an area of about 190 mm×280 mm except for the fringing blank portion is the size of the viewing image. Accordingly, in this invention, in addition to the “size of the viewing image” and the “size of the main photographic object in the viewing image”, the “size of the output medium” described in the structures (22) to (24) is also an adjustment factor.  
     [0192] The above-mentioned “size of the main photographic object in the viewing image” means the actual size of the area defining the main photographic object when it is printed or displayed on an output medium; for example, even in cases where images are printed on a paper of the same A4 size, in a close-up photograph of a person, the size becomes of the order 50 mm to 100 mm square in many cases, and in a photograph of assembling persons or in a memorial photograph of a person with a background of a building, the size becomes of the order 10 mm to 30 mm in many cases. In this way, even if the size of the viewing image is supposed to be constant, the size of the main photographic object in the viewing image becomes different for each image, depending on the composition of the photograph intended by the photographing person.  
     [0193] As regards the method of obtaining the above-mentioned “size of the viewing image”, in the case where the mode of practice of this invention is an image recording apparatus or an image processing apparatus connected to a printer, the size of the viewing image can be easily obtained from the content of print instruction, and also in the case where the mode of practice of this invention is an image processing method, or an image processing apparatus in which a printer is not directly connected and image data are outputted to a file or a communication means, by a user designating the size of the supposed viewing image, it becomes possible for the above-mentioned mode of practice of this invention to obtain the size of the viewing image.  
     [0194] Further, even in a situation where the size of the viewing image cannot be obtained, by “estimating the size of the viewing image”, it becomes possible to practice this invention. For example, the number of pixels of a viewing image strongly correlates with the size of the image on an output medium intended by the user. It is inferred that an image of the order 10,000 pixels is used for a thumbnail index, and the size of the viewing image is intended to be of an order of several cm square, and an image of the order of 2,000,000 pixels is intended to be a print of L-2L size. The simplest way of estimating the size of a viewing image is a method such that the number of pixels of the viewing image is converted into the size of the viewing image by assuming a general print condition of a color image (300 DPI for example).  
     [0195] As regards the method of obtaining the above-mentioned “size of the main photographic object in the viewing image”, the acquisition of the size is accomplished by receiving a designation when a user gives an instruction to a mode of this invention, or by referring to the supplementary information (what is called tag information, etc.) attached to the image signal representing the photographed image. As regards the method of attaching such supplementary information, it is appropriate to give it as an independent information format, or an independent information file/signal; however, for a more desirable mode of practice, to utilize already existing tag information defined in various kinds of versatile image format represented by JPEG, TIFF, Exif, etc., or to utilize the area which is freely available such as a maker-note or a user-note can be cited.  
     [0196] It is unnecessary that the size of the main photographic object is specified as an actual size on the viewing image, and it is also appropriate to give information to make it possible to calculate the “proportion occupied by the main photographic object in the image” by a similar means as described above. For example, in the case where the size of the main photographic object is given as the number of pixels on one side or on the radius, by comparing it with the total number of pixels of the image as a whole, the proportion occupied by the main photographic object in the image is easily calculated, and by multiplying the size of the viewing image which has been obtained or estimated by the above-mentioned method in this invention, the size of the main photographic object in the viewing image can be easily calculated.  
     [0197] Further, in a situation where the size of the main photographic object is not directly specified, one can practice this invention by estimating the “size of the main photographic object in the viewing image”. To state it concretely, a method to extract the portion of skin color from the image signal, or a method to extract the range in the central part of an image where hue and lightness do not exceed a specified range can be cited; however, for a more desirable method, a method in which information on the photographing conditions is obtained and from this information, the proportion occupied by the main photographic object is estimated can be cited. For a concrete example of the information on the photographing conditions, the area of photographic object, the type of photographing scene, object distance, detection of strobe light reflection, etc. can be cited, and these kinds of information may be given as an independent information format or an independent information file/signal; however, for a more desirable mode of practice, to utilize already existing tag information defined in various kinds of versatile image format represented by JPEG, TIFF, Exif, etc., or to utilize the area which is freely available such as a maker-note or a user-note can be cited.  
     [0198] The method to estimate the proportion occupied by the main photographic object from the information on the photographing conditions will be explained with reference to examples.  
     [0199] For example, in the case where information of photographic object area is given, if this information is provided in a form to make it possible to calculate the number of pixels in the area, that is, in a form of the central coordinates of the circular area and the number of pixels on the radius, or in a form of the central coordinates of the rectangular area and the number of pixels of the longitudinal and lateral sides, by comparing the number of pixels within the area with the total number of pixels of the image as a whole, one can easily calculate the proportion occupied by the main photographic object in the image, and by multiplying the size of the viewing image which has been obtained or estimated by the above-mentioned method of this invention, one can easily calculate the size of the main photographic object in the viewing image. Further, in the case where the information of the photographic object area is given only as a point at the central coordinates of the area, by calculating the average hue and average lightness of several pixels around the specified central coordinates, and searching a pixel area around the specified central coordinates where the hue and the lightness do not exceed a specified range from the above-mentioned average values (for example, the difference in hue angle is within 10 degree, and the difference in lightness is within 10% of the dynamic range), one can determine the number of pixels of the main photographic object area.  
     [0200] Further, in the case where information of the type of the photographed scene is given, if it is a photograph of a person or a portrait for example, it is inferred that the main photographic object is the face of the person, and the diameter is of the order ¼ to ½ of the shorter side of the image. The proportion occupied by the main photographic object can be estimated as an approximate value by using the number of pixels equivalent to the area of a circle of a diameter having the number of pixels of the order ⅓ of that of the shorter side of the image, and the accuracy of the estimation can be also improved by searching a skin-colored area of the order ¼ to ½ of the shorter side of the image from the inside area of the image. Further, in the case where the type of the photographed scene is landscape, there exists in most cases a main photographic object occupying the area of about 10% of the image area. Further, in the case where the type of photographed scene is a night scene, it can be inferred that an area having higher lightness values compared to the image as a whole is the main photographic object. Further, in the case where information of the object distance is given, supposing that the main photographic object is a human face, by calculating the size of a human face corresponding to the object distance from the lens characteristic of general compact cameras, one can estimate the proportion occupied by the main photographic object in the image. Further, in the case where information of the strobe reflection light is given, an approximate object distance can be estimated by the presence or absence and the strength of the reflection light, and the proportion occupied by the main photographic object in the image can be estimated by the same way as described in the above. It is a matter of course that, in the case where a plurality of kinds of such photographing information are given, the accuracy of estimation can be improved by the judgment based on the combination of them.  
     [0201] This invention is characterized by the content of an image processing to be applied to image data representing a photographed image being determined in order that a good impression may be given to a person observing the image on an output medium on the basis of the “size of the viewing image”, and the “size of the main photographic object in the viewing image” obtained or estimated by the above-mentioned method. The concrete mode of practice of this image processing will be explained.  
     [0202] When a person observes a photographed image printed or displayed on an output medium of a specified size, its general observation distance changes with the “size of the output medium” and the “size of the viewing image”. The observation distance in the case of size of the image being L-size (86 mm×128 mm) to 2L-size (128 mm×172 mm) is of the order 30 cm, and in the case where the image is larger than this, the observation distance becomes longer in order to make the whole image come within the field of sight. This distance mostly becomes a distance such that the longer side of the image comes within the angle of visual field about 30°. On the contrary, even in the case where the image is small, it never happens that the observation distance comes greatly nearer than 30 cm. This relation is shown in FIG. 1. Besides, a photographed image referred to herein means what is called a photographic image photographed by a means such as a film camera or a digital camera. For example, in the case where an illustration of a frame is drawn on an output medium, and with this illustration used as a template, a photographed image is inserted inside it, the observation distance is generally dominated not by the size of the illustration of the frame but by the size of the photographed image inside the frame. Accordingly, the “size of the viewing image” in this invention means the size of the photographed image which is the object of observation inside the template. Further, in the case where a plurality of images having different sizes respectively are inserted inside one template, the size of the viewing image is defined independently for each photographed image.  
     [0203] On the other hand, the impression given to an observer from an image depends on how the main photographic object appears at the above-mentioned observation distance. It is generally said that resolution in the human visual sensation is high within the range of the angle of visual field 10°, and for example, in the case where the observation distance is 30 cm, the range defined by a circle of a diameter of 5 cm is equivalent to the range of the angle of visual field 10°. The angle of visual field is defined by the following expression.  
     [0204] [M1} 
     θ=2 tan −1 ( d/r )  
     [0205] (θ: angle of visual field, d: radius of main photographic object, r: observation distance).  
     [0206] In the case where the size of the main photographic object in an image is smaller than the range of the angle of visual field 10°, a sharp picture with a rather strong contrast and a clear-cut contour is generally preferred. For example, if a human face is the main photographic object and the diameter is small as not greater than 5 cm, a photograph of clearly marked features is preferred. Especially, as in the case of a photograph of assembling persons, if the diameter of the face is of the order 1 cm to 2 cm, a high-contrast picture such that persons are easy to recognize is preferred. On the contrary, in the case where the main photographic object is larger than the range of the angle of visual field 10°, a picture with a low contrast giving a soft impression is generally preferred. For example, in the case where a human face is the main photographic object and the diameter is 10 cm, the skin of a part of the face come to be within the range of the angle of visual field 10°. If the contrast is too strong or the sharpness is too strong in this situation, the picture gives an impression that the skin is rough, gives an impression that the shade around the eyes and noses is too strong, or makes an observer have a feeling of disorder in the variation of chroma at detailed portions. In this case, if the contrast is made softer, and the sharpness and the variation of chroma are made rather suppressed, it gives an impression that the skin is smooth and fresh.  
     [0207] It is a desirable mode of a method for determining the content of the image processing to be applied to image data representing a photographed image that, on the basis of the above-mentioned empirical rule, the observation distance is estimated from the “size of the output medium”, and the “size of the viewing image”, the angle of visual field of the main photographic object as a whole is estimated from this observation distance and the “size of the main photographic object in the viewing image”, and in accordance with this angle of visual field of the main photographic object as a whole, the setting of the contrast/degree of sharpness enhancement/degree of granular noise suppression/degree of chroma enhancement of an image are controlled. As the result, it becomes possible that an image being observed on an output medium gives a more desirable impression. Besides, with a granular noise taken for instance as the object of smoothing processing, the processing is referred to as granular noise suppression, and the degree is referred to as the degree of granular noise suppression. Further, the degree of sharpening processing is referred to as the degree of sharpening enhancement. Examples of these are shown in FIG. 2 to FIG. 5. Further, in the case where a technology using the suppression of middle frequency component and the enhancement of high frequency component together is used for the method of sharpness enhancement and granular noise suppression, it is desirable to control the settings of the suppression of middle frequency component and the enhancement of high frequency component in accordance with the above-mentioned angle of visual field of the main photographic object as a whole. Examples of these are shown in FIG. 6 and FIG. 7. The middle frequency component is the component of a spatial frequency band containing granularity in a color image, and the high frequency component is the component of a spatial frequency band containing edges or textures in a color image.  
     [0208] In addition, for the sake of explanation, from the “size of the output medium”, the “size of the viewing image”, and the “size of the main photographic object in the viewing image”, the observation distance and the angle of visual field of the main photographic object as a whole are calculated, and the image processing conditions are determined; however, in the actual mode of practice, it is unnecessary to carry out the above-mentioned calculation every time image data representing a photographed image are received. It is desirable to determine image processing conditions with reference to a look-up table in which image processing conditions in accordance with the “size of the output medium”, the “size of the viewing image”, and the “size of the main photographic object in the viewing image” are noted beforehand.  
     [0209] Further, in this invention, it is desirable that the determination of image processing conditions through the calculation of the above-mentioned angle of visual field, and the determination of image processing conditions with reference to a look-up table having image processing conditions noted beforehand or an expression of simple approximate calculation are carried out for every kind of output medium as described in the structures (19) to (21).  
     [0210] Further, in this invention, it is desirable that the tendency of the image processings which have been practiced when the image data representing a photographed image were generated is grasped, and on the basis of the information, the degree of the above-mentioned image processings is modified. To state it concretely, for example, it is desirable that the tendency of image processings such as a contrast processing, a sharpness enhancement, and a chroma enhancement which have been applied to image data when they were generated is grasped, and on the basis of the information, the degree of the above-mentioned image processings are modified. As regards the method of obtaining the information concerning such tendency of image processings, it is appropriate to obtain the information as an independent information format or an independent information file/signals; however, for a more desirable mode of practice, referring to existent tag information defined in various kinds of versatile image formats represented by JPEG, TIFF, Exif, etc., and utilizing a freely available area such as a maker-note, or a user-note. On the basis of such information, supposing that contrast has been already set at a hard tone at the time of photographing, the image characteristics of the viewing image can be made to fall within a proper range through modifying the degree of image processings in this invention in a way such that the duplication/reciprocity with the image processings at the time of photographing is taken into consideration; that is, the contrast enhancement in the case of the size of the main photographic object on the image being smaller than the range of the angle of visual field 10° is suppressed or canceled, and the degree of contrast softening in the case of the main photographic object on the image being larger than the range of the angle of visual field 10° is made large.  
     [0211] The expression “the amount of application of said sharpening processing being adjusted in accordance with the photographed scene” described in the structures (28), (29), and (30) means that, as the result of the discrimination of the photographed scene, the amount of edge enhancement by means of an unsharp mask, for example, is adjusted in accordance with the kind of the main photographic object or the composition of the photographed scene, for example. For an artificial photographic object such as a building which is composed of many straight lines and easy to make a border with the sky, a more feeling of sharpness is desired, but on the contrary, for a living thing such as a person or a flower, a rather weak setting of sharpness is preferred. In accordance with the photographed scene, or more desirably, by a partial adjustment of the amount of the application of sharpening processing, it is possible to obtain a more subjectively desirable image quality.  
     [0212] It is more desirable in practicing this invention to use a combination of the processings set forth respectively in the structures (19) to (21), (22) to (24), (25) to (27), and (28) to (30) in a composite way.  
     [0213] Further, as regards the sharpening processing of this invention, it is possible to use a method such that digital image data are divided into luminance information and color information, and sharpness processing is applied to the luminance information only. By the application of sharpening processing to the luminance information only using this method, image sharpness can be raised without being accompanied by the deterioration of the color information.  
     [0214] An “image processing apparatus” of this invention is equipped with means for transforming scene-referred raw data outputted by an image-capturing device depending on the characteristics of the image-capturing device into standardized scene-referred image data and standardized viewing image referred image data.  
     [0215] An “image recording apparatus” of this invention is an apparatus such that means for forming a viewing image on an output medium is attached to an image processing apparatus.  
     [0216] An image processing apparatus and an image recording apparatus of this invention may be equipped with, in addition to a mechanism for applying image processings of this invention to image data obtained by an image-capturing device relating to this invention, a film scanner for inputting frame image information recorded by an analog camera on a photographic photosensitive material such as a color negative film, a color reversal film, a black-and-white negative film, or a black-and-white reversal film, or a flat-bed scanner for inputting image information reproduced on a color photographic paper, which is a silver halide photographic paper. Further, they may be equipped with means for reading digital image data which are obtained by a digital camera other than an image-capturing device relating to this invention and stored in any one of portable “media” known to the public such as a compact flash (registered trade mark), a memory stick, a smart medium, a multi-media card, a floppy (registered trade mark) disk, an opto-magnetic storage medium, or a CD-R, or a processing means for obtaining digital image data from a remote site through a communication means such as a network and forming a viewing image on any one of “recording media” known to the public such as a display device such as a CRT, a liquid crystal display, or a plasma display, or a paper material for the production of a hard copy image such as a silver halide photographic paper, an ink jet paper, or a thermal printer paper.  
     [0217] The file structures of digital image data to be used in this invention are shown respectively in FIG. 8 to FIG. 13.  
     [0218] As shown in FIG. 8, FIG. 10, and FIG. 12, correction data for image-capturing device characteristics are attached to scene-referred raw data, scene-referred image data, and viewing image referred image data.  
     [0219] Further, as shown in FIG. 12 and FIG. 13, the difference data of scene-referred image data and viewing image referred image data are attached to viewing image referred image data.  
     [0220] Moreover, as shown in FIG. 9, FIG. 11, and FIG. 13, it is desirable that photographing information data are further attached to scene-referred raw data, scene-referred image data, and viewing image referred image data.  
     [0221] The term “photographing information data” means a record of the setting of photographing conditions at the time of photographing, and it may include the same one as tag information to be written at the header part of an Exif file. To state it concretely, it is, for example, a tag (code) indicating information concerning an exposure time, a shutter speed, an aperture value (f-number), an ISO sensitivity, a luminance value, an object distance range, a light source, strobe light emitted or not, an object region, white balance, a zoom magnification, an object structure, a photographed scene type, a reflection light quantity of strobe light, photographed chroma, and a kind of a photographic object, or the like.  
     [0222] The above-mentioned “photographing information data” are classified into values obtained by a sensor provided in a camera at the time of photographing for the purpose of automating the exposure setting and focusing function in the camera, data obtained by processing the above-mentioned values outputted by the sensor, and photographing conditions of the camera set on the basis of said sensor; however, in addition to these, information manually set by a photographing person concerning the photographing mode dial (for example, portrait mode, sports mode, macro-photographing mode, etc.) and the setting switch for enforced strobe light emission is included.  
     [0223] Examples of the optimization of viewing image referred image data using the above-mentioned “photographing information data” will be shown below.  
     [0224] By the “object structure” information, it becomes possible to partially apply a chroma enhancement processing for example, or to apply a dodging processing for a scene having a broad dynamic range. By the “photographed scene type” information, for example in a night scene photograph, it becomes possible that the degree of white balance adjustment is lowered, and color balance is specially adjusted. By the “reflection light quantity of strobe light” information, the distance between the photographing person and the object can be estimated, and it becomes possible to reflect the information to the condition setting for the image processing to suppress the whitening of skin part for example. By the “kind of a photographic object” information, for the photographing of a person for example, it becomes possible to make the creases of skin unnoticeable by lowering the degree of sharpening processing and strengthening the smoothing processing.  
     [0225] Further, for the purpose of supplementing the “photographing information data” such as the “object structure”, the “photographed scene type”, the “reflection light quantity of strobe light”, or the “kind of photographic object”, the information such as the “exposure time”, “shutter speed”, “aperture value (f-number)”, “ISO sensitivity”, “luminance value”, “object distance range”, “light source”, “strobe light emitted or not”, “object region”, “white balance”, and “zoom magnification” can be used in a supplementary way. Further, it is possible that the amount of application of noise suppression processing is adjusted from the “ISO sensitivity” information, or the “light source” information is used for re-adjustment of white balance.  
     [0226] In the mode of practice where the above-mentioned “photographing information data” are stored in a medium independently of the “scene-referred raw data”, the “scene-referred image data”, or the “viewing image referred image data”, it is necessary that, to either the “photographing information data” or any one of the “scene-referred raw data”, the “scene-referred image data”, and the “viewing image referred image data” or to both of them, information for correlating both is attached or a status information file having relating information noted is attached separately.  
     [0227] In the case where it is required a processing to transform “viewing image referred image data” into “scene-referred image data” using an “image-capturing device” of a type to generate the above-mentioned viewing image referred image data” in an image forming method, an image processing apparatus, and an image recording apparatus, it is desirable that image processing of this invention is composed of at least a first step for practicing a correction for the transformation characteristics specific to the image-capturing device through correcting at least one of “gradation characteristic”, “color reproduction characteristic”, and “frequency characteristic” to reproduce a state where no difference is present for each image-capturing device, a second step for practicing an exposure control processing and a gray balance processing, and a third step for practicing a gradation transformation processing.  
     [0228] In addition, in the case where an image-capturing device of a type to generate “scene-referred image data” is used, the above-mentioned first step is omitted. In the case where an image-capturing device of a type to output “scene-referred raw data” depending on the image-capturing device characteristic to be used in generating “scene-referred image data” and “reproduction assisting data in practicing correction processing for image-capturing device characteristics”, the generation process of the “scene-referred image data” corresponds to the above-mentioned first step.  
     [0229] The above-mentioned “step” defines the processing flow of an image forming method of this invention, and the processing proceeds successively from the first step to the second step. By separating the processing into the steps in this way, further improvement of accuracy and higher efficiency can be achieved.  
     [0230] Further, it is desirable that the gradation alteration processing in the above-mentioned third step includes a non-linear correction processing, and further, it is desirable that the above-mentioned third step is composed of two steps, namely, a step for correcting γ and a step for practicing a non-linear correction, and it has a structure such that γ correction is done first.  
     [0231] The above-mentioned first step is a transformation process for transforming “scene-referred raw data” into “scene-referred image data”, or transforming “viewing image referred image data” into “scene-referred image data”, and the second step has both the aspect of a transformation process into “scene-referred image data” through removing the difference in the characteristic specific to the image-capturing device to achieve the equalization of image data and the aspect of a transformation process into “viewing image referred image data” for making image quality more subjectively desirable on an output medium, and carries out the bridging between the first step and the third step. The third step is a transformation process into viewing image referred image data for making image quality more subjectively desirable on an output medium.  
     [0232] The term “exposure control processing” means a processing to correct the brightness of captured-image data as a whole. Although image data are corrected for the representation of object information specific to the image-capturing device and equalized, brightness is different between image-capturing device owing to the performance of automatic exposure control (hereinafter referred to also as “AE”) of image-capturing device dispersing from one to another and the difference in the manual setting by the photographing persons.  
     [0233] The term “gray balance adjustment processing” means a processing to correct color bias of image data as a whole. Although image data are corrected for the representation of object information specific to the image-capturing device and equalized, color bias is different between image-capturing device owing to the performance of automatic gray balance control (hereinafter referred to also as “AWB”) of image-capturing device dispersing from one to another and the difference in the manual setting by the photographing persons.  
     [0234] It is desirable to make the mode of practice such one that exposure control (AE) processing and gray balance adjustment (AWB) processing in the above-mentioned second step are carried out parallel by means of an input-to-output conversion table (look-up table (hereinafter referred to also as an “LUT”)) set for each of the colors B, G, and R. To state it concretely, gray balance is adjusted by the parallel moving of the respective LUT&#39;s of the colors B, G, and R done individually, and brightness is adjusted by the parallel moving of the LUT&#39;s done altogether.  
     [0235] The term “gradation alteration processing” is a processing to adjust gradation so as to become more subjectively desirable on an output medium, that is, on a display device such as a CRT, a liquid crystal display, and a plasma display, or on a paper material for hard copy image production such as a silver halide photographic paper, an ink jet paper, and a thermal printer paper. To state it concretely, “non-linear processing” and “γ correction processing” described below is included.  
     [0236] The term “non-linear correction processing” means a processing to suppress whitening at highlight portions and fill-in at shadow portions of an image resulting from the density reproduction range of the output medium being narrow by non-linearly correcting highlight portions and shadow portions of an image. The amount of correction is obtained in the following way and is reflected to the corrected curve: That is, in such a case that print density is raised by the above-mentioned second step (AE/AWB processing), gradation in the highlight side is made harder, while the gradation in the shadow side is made softer, and in such a case that conversely print density is lowered, gradation in the highlight side is made softer, while the gradation in the shadow side is made harder.  
     [0237] The term “to correct γ” means to correct a gradation curve to become a reference gradation curve set for each output medium. In this specification, for example, in the case where the output medium is a silver halide photographic paper, the reference gradation curve is an S-shaped curve, and its middle part is set to have γ=1.6. A processing having a purpose to make better the visual appearance of a printer output by making the gradation harder is called a “γ correction processing”.  
     [0238] In the following, a process to practice a smoothing processing of this invention will be described in detail.  
     [0239] The term “a scene-referred image data generation process” in the description of the structures (1) and (2) means the above-mentioned first step. Further, the expression “said scene-referred image data generation means comprising a smoothing processing means for applying a smoothing processing to said captured-image data” in the description of the structures (6), (7), (11), and (12) means that a smoothing processing is practiced in the above-mentioned “first step”. This invention is characterized by it that a smoothing processing is practiced in the “first step”, that is, in the transformation process to transform “scene-referred raw data” into “scene-referred image data” or to transform “viewing image referred image data” into “scene-referred image data”. That is, a “noise” to become the object of the smoothing processing here results from the image-capturing device. Because the noise is different for each image-capturing device and specific to it, the information volume of digital image data (for example, the number of gray levels, the number of pixels, etc.) to become the object of processing is large, and the cause of noise generation is clearer in the scene-referred image data generation process, it becomes possible to prepare a more appropriate smoothing processing means; as the result, it is possible to practice a smoothing processing with an artifact suppressed.  
     [0240] In the description of the structures (4), (5), (9), (10), (14), and (15), a smoothing processing is applied to “generated scene-referred image data”. That is, after the above-mentioned “first step”, a smoothing process is practiced in the pre-processing of “the second, and third steps”. In the case where an image-capturing device of a type to generate “scene-referred image data” by image pickup is used, the above-mentioned “first step” is omitted, but a smoothing processing is carried out after that; therefore, even in the case where an image-capturing device of a type to generate “scene-referred image data according to this invention, this invention has the merit that it can lighten the processing load of the image-capturing device, because there is no need of practicing a smoothing processing in the image-capturing device.  
     [0241] In the following, a process to practice a sharpening processing of this invention will be described in detail.  
     [0242] The term “viewing image referred image data generation process” in the description of the structures (1) and (4) means the above-mentioned “third step”. Further, the expression “said viewing image referred image data generation means comprising a sharpening processing means for applying a sharpening processing to said scene-referred image data” in the description of the structures (6) and (11) and the expression “said viewing image referred image data generation means comprising a sharpening processing means for applying a sharpening processing to said scene-referred image data having been subjected to said smoothing processing” in the description of the structures (9) and (14), mean that a sharpening processing is practiced in the above-mentioned “second or third step”. This invention is characterized by it that a sharpening processing is practiced in the transformation process into viewing image referred image data to make the image quality subjectively more desirable on an output medium, for example. That is, in this description, a sharpening processing is optimized, for example, in accordance with the kind of the output device, output medium, in particular, with the size of the output medium; as the result, it is obvious that a more subjectively desirable image quality can be obtained.  
     [0243] According to the description in the structures (2), (5), (7), (10), (12), and (15), a sharpening processing is applied to the “generated viewing image referred image data”. That is, a sharpening processing is practiced in a process in a further later process of the above-mentioned “second or third step”. In this invention, in a case such that, in a situation where the same output device or an output medium of the same kind is used for example, only the size of the output medium is changed, the modification in the above-mentioned “second or third step” is not required, and in such a case, the “second or third step” can be omitted; therefore, this invention has the merit that the load of processing can be lightened.  
     [0244] The expression “a smoothing processing is practiced at first and a sharpening processing is practiced next in said scene-referred image data generation process” means that a smoothing processing is practiced at first and a sharpening processing is practiced next in the above-mentioned “second or third step”. Further, a viewing image referred image data generation means described in the structures (8) and (13) practices a smoothing processing at first and practices a sharpening processing next in the “viewing image referred image data generation process”, that is, in the above-mentioned “second or third step”. In the case where an image-capturing device of a type to generate “scene-referred image data” by image pickup is used, the “first step” is omitted, but a smoothing processing and a sharpening processing is practiced in the “second or third step”; therefore, this invention has the merit that even if an image-capturing device of a type to generate “scene-referred image data” is used, it is made unnecessary by this invention to practice a smoothing processing in the image-capturing device, which makes it possible to lighten the load of processing in the image-capturing device. Further, because it is put into practice to practice a smoothing processing and a sharpening processing in the “second or third step”, the speed of processing is made higher.  
     EMBODIMENT OF THE INVENTION  
     [0245] Embodiment 1  
     [0246] In the following, the embodiment 1 of this invention will be explained on the basis of the drawings.  
     [0247] In addition, in the embodiment 1, it is shown an example in which an “image-capturing device” of a type to generate “scene-referred raw data” depending on the characteristic of the image-capturing device for use in the generation of the above-mentioned “scene-referred image data”, “reproduction assisting data in practicing a correction processing for the image-capturing device characteristics” (hereinafter referred to as “correction data for the image-capturing device characteristics”), and “photographing information data”. In the following, an image-capturing device  21 , which is an “image-capturing device” of a type to output “scene-referred raw data”, “correction data for image-capturing device characteristics”, and “photographing information data” will be explained.  
     Structure of Image-Caputuring Device  21   
     [0248] First, the structure will be explained.  
     [0249]FIG. 14 is a block diagram showing the functional structure of an image-capturing device  21  relating to this invention. As shown in FIG. 14, the image-capturing device  21  has a structure equipped with a lens  1 , a diaphragm  2 , a CCD  3 , an analog processing circuit  4 , an A/D converter  5 , a temporary storage  6 , an image processor  7 , a header information processor  8 , a storage device  9 , a CCD drive circuit  10 , a controller  11 , a photographing information data processor  12 , a correction information processor for apparatus characteristics  13 , an operation section  14 , a display section  15 , a strobe drive circuit  16 , a strobe light  17 , a focal distance adjusting circuit  18 , an auto-focus drive circuit  19 , a motor  20 , etc.  
     [0250] The optical system of the image-capturing device  21  has a structure equipped with the lens  1 , the diaphragm  2 , and the CCD (solid state image sensor)  3 .  
     [0251] The lens  1  has a function of focusing, to form an optical image of a photographic object. The diaphragm  2  adjusts the light quantity of a bundle of light having passed the lens  1 . The CCD  3  photoelectrically converts the light image formed on the light receiving surface by the lens  1  into electrical signals (pickup image signals) corresponding to the incident light quantity for each pixel in the CCD  3 . Further, the CCD  3 , being controlled by timing pulses outputted from the CCD drive circuit  10 , outputs these pickup image signals to the analog processing circuit  4  successively.  
     [0252] The analog processing circuit  4  applies an amplification processing of R, G, and B signals, noise reduction processing, etc. to the pickup image signals inputted from the CCD  3 . As regards these processings in the analog processing circuit  4 , ON/OFF switching can be made through the controller  11  in response to an operation signal from the operation section  14 .  
     [0253] The A/D converter  5  converts the image pickup signals inputted from the analog processing circuit  4  into digital image data and outputs them.  
     [0254] The temporary storage  6  is a buffer storage or the like, and temporarily stores the digital image data outputted from the A/D converter  5 .  
     [0255] The image processor  7  applies processings such as gradation correction, cross-talk correction for the spectral sensitivity, dark current noise suppression, sharpening, white balance adjustment, chroma adjustment, and in addition to these, change of image sizes, trimming, aspect transformation, etc. to the digital image data for use in the display in the display section  15 . As regards these processings in the image processor  7 , ON/OFF switching can be made through the controller  11  in response to an operation signal from the operation section  14 .  
     [0256] The header information processor  8  applies a processing to write correction data for the image-capturing device characteristics d 1  generated by the correction information processor for apparatus characteristics  13  as header information to the digital image data stored in the temporary storage  6 .  
     [0257] The storage device  9  is made up of a non-volatile semiconductor storage device, etc., and is composed of a recording medium such as a memory card for recording photographed digital image data and a storage capable of being read having a control program for the image-capturing device  21  stored.  
     [0258] The CCD drive circuit  10  outputs timing pulses on the basis of a control signal outputted from the controller  11 , to carry out the drive control of the CCD  3 .  
     [0259] The controller  11  is made up of a CPU (Central Processing Unit), etc.; it reads out a control program for the image-capturing device  21  stored in the storage device  9 , and carries out the control of the whole image-capturing device  21  in accordance with the program read out. To state it concretely, the controller  11 , in response to an operation signal from the operation section  14 , executes a control of the auto-focus drive circuit  19  practicing a control of the motor  20  for adjusting the focal distance and the focusing of the lens  1 , the focal distance adjusting circuit  18 , the CCD drive circuit  10 , the analog processing circuit  4 , the temporary storage  6 , the image processor  7 , the operation section  14 , the display section  15 , the strobe drive circuit  16 , and the correction information processor for apparatus characteristics  13 , to carry out photographing.  
     [0260] When the output of scene-referred raw data is instructed by the operation section  14 , the controller  11  carries out a scene-referred raw data saving processing to be described later, omitting the signal amplification and noise reduction processing in the analog processing circuit  4  and the processings in the image processor  7 , writes correction data for the image-capturing device characteristics d 1  and photographing information data d 3  in a recording medium of the storage device  9  as header information, while it records photographed image data as scene-referred raw data d 2 .  
     [0261] The photographing information data processor  12  generates photographing information data d 3 . The photographing information data d 3  is, for example, information directly relating to the kind of a camera (model) such as a camera name or a code number, information concerning the setting of photographing conditions such as an exposure time, a shutter speed, an aperture value (f-number), an ISO sensitivity, a luminance value, an object distance range, a light source, strobe light emitted or not, an object region, white balance, a zoom magnification, an object structure, a photographed scene type, a reflection light quantity of strobe light, photographing chroma, information concerning the kind of a photographic object, or the like.  
     [0262] The correction information processor for apparatus characteristics  13 , in the case where scene-referred raw data which are not subjected to processings in the analog processing circuit  4  and image processor  7  under the control of the controller  11  are recorded in a recording medium of the storage device  9 , generates correction data for image-capturing device characteristics d 1  as information necessary for transforming the scene-referred raw data d 2  into scene-referred image data d 4  in a standardized color space such as RIMM RGB, or ERIMM RGB, and outputs them to the header information processor  8 . The correction data for image-capturing device characteristics d 1  are equivalent to the “reproduction assisting data in practicing a correction processing for image-capturing device characteristics to generate standardized scene-referred image data” described in the claims of this invention.  
     [0263] The operation section  14  is provided with various kinds of functional button (not shown in the drawing) such as a release button, an ON/OFF switching button for the power source, and a zoom button, a cursor key, etc., and outputs operation signals respectively corresponding to the buttons and the keys as control signals to the controller  11 . In this embodiment of the invention, the operation section  14  is equipped with a functional button for instructing the output of scene-referred raw data.  
     [0264] The display section  15  displays digital image data by a control signal from the controller  11 , while it displays information for a user of the image-capturing device  21  confirming the settings and conditions concerning photographing.  
     [0265] The strobe drive circuit  16  controls the strobe  17  to drive it when the brightness of a photographic object is low by a control signal from the controller  11  and makes the strobe emit light.  
     [0266] The strobe  17  has its battery voltage raised to a specified high voltage and accumulates electric charge in a condenser. Then, driven by the strobe drive circuit  16 , it emits light from the X-tube by the high voltage due to the electric charge stored in the condenser, to irradiate the photographic object by assistant light.  
     [0267] The focal distance adjusting circuit  18  moves the lens  1  by a control signal from the controller  11  to practice a control of the motor  20  for adjusting the focal distance.  
     [0268] The auto-focus drive circuit  19  moves the lens  1  by a control signal from the controller  11  to practice a control of the motor  20  for adjusting the focus.  
     Operation of image-Capturing Device  21   
     [0269] In the following, the operation will be explained.  
     [0270]FIG. 15 is a flow chart showing a scene-referred raw data saving processing to be practiced by a control of the controller  11  when the output of photographed digital image data as scene-referred raw data is set by the operation section  14 , and the release switch is depressed. In the following, scene-referred raw data saving processing will be explained with reference to FIG. 15.  
     [0271] When the release button of the operation section  14  is depressed, the controller  11  controls the pertinent sections to practice photographing (step S 11 ). The pickup image signals obtained from the CCD  3  are converted into digital image data by the A/D converter  5 , to generate scene-referred raw data (step S 12 ). Further, correction data for image-capturing device characteristics d 1  are generated by the correction information processor for apparatus characteristics  13  (step  13 ), and photographing information data d 3  are generated by the photographing information data processor  12  (step S 14 ).  
     [0272] After the generation of scene-referred raw data d 2 , correction data for image-capturing device characteristics d 1 , and photographing information data d 3 , the correction data for image-capturing device characteristics d 1  and the photographing information data d 3  are recorded and attached at the file header of the scene-referred raw data d 2  (step S 15 ), a data file having both the data already attached is prepared (step S 16 ), and this data file having both the data attached is recorded and saved in a recording medium of the storage device  9  which has such a structure as to be capable of being mounted to and dismounted from the image-capturing device  21  (step S 17 ).  
     [0273]FIG. 9 is a drawing showing the data structure of digital image data to be recorded in a recording medium of the storage device  9  at the step S 17 . As shown in FIG. 9, digital image data obtained by photographing are recorded as scene-referred raw data d 2 , and at the header area of it, correction data for image-capturing device characteristics d 1  and photographing information data d 3  are recorded. By taking out this recording medium from the image-capturing device  21 , and loading it in an external apparatus such as an image processing apparatus, or an image recording apparatus, it is possible to output scene-referred raw data d 2 , correction data for image-capturing device characteristics d 1 , and photographing information data d 3  to the above-mentioned external apparatus.  
     Structure of Image Processing Apparatus  117   
     [0274] Next, the embodiment of an image processing apparatus of this invention will be explained.  
     [0275] First, the structure will be explained.  
     [0276]FIG. 16 is a block diagram showing the functional structure of an image processing apparatus  117  of this invention. As shown in FIG. 16, the image processing apparatus  117  consists of an input section  101 , a header information analysis section  102 , a correction processor for image-capturing device characteristics  115  for applying a correction processing for image-capturing device characteristics to scene-referred raw data d 2  on the basis of correction data for image-capturing device characteristics d 1  to generate scene-referred image data d 4 , and an optimization processor  116  for applying an optimization processing to the scene-referred image data d 4  generated by the correction processor for image-capturing device characteristics  115  to generate viewing image referred image data d 5 . The header information analysis section  102  is connected to each of the correction processor for image-capturing device characteristics  115  and the optimization processor  116 , and further, the optimization processor  116  is in a state of being capable of connection to each of a storage device  110 , an output device  111 , and a display device  112 . The above-mentioned structural components operate under the presiding control of a controller  118  made up of a CPU, etc.  
     [0277] The input section  101  is equipped with a recording medium loading section (not shown in the drawing) when a recording medium having a file (refer to FIG. 9) of digital image data (captured-image data) obtained through photographing by the above-mentioned image-capturing device  21  recorded on it is loaded in this loading section, the input section  101  reads out the data file having image data recorded to output it to the header information analysis section  102 . Besides, in this embodiment of the invention, the input section  101  is explained as reads out data from a recording medium loaded; however, it is also appropriate that a data communication cable or a wireless or wired communication means is provided, and data are inputted through any one of these communication means.  
     [0278] The header information analysis section  102  analyzes data inputted from the input section  101 , divides the data into scene-referred raw data d 2 , correction data for image-capturing device characteristics d 1 , and photographing information data d 3 , and outputs the correction data for image-capturing device characteristics d 1  to a correction processor for apparatus characteristics  103   a , the scene-referred raw data d 2  to a smoothing processor  113 , and the photographing information data to a photographing data processor  106 .  
     [0279] As shown in FIG. 16, the correction processor for pickup image apparatus characteristics  115  has a structure comprising the correction processor for apparatus characteristics  103   a , a processing condition table  103   b , the smoothing processor  113 , a scene-referred image data generator  104 , and a temporary storage  105 . the correction processor for pickup image apparatus characteristics  115  has a function as a scene-referred image data generation means of an image processing apparatus as set forth in the claims of this invention.  
     [0280] When correction data for image-capturing device characteristics d 1  are inputted from the header information analysis section  102 , the correction processor for apparatus characteristics  103   a  determines the generation condition of scene-referred image data d 4  by referring to the processing condition table  103   b . The processing condition table  103   b  is a table storing processing conditions for generating scene-referred image data d 4  in correspondence to each of the image-capturing device characteristics.  
     [0281] The smoothing processor  113  applies, for example, an orthogonal transformation such as a DCT (Discrete Cosine Transformation), a Fourier transformation, or a multi-resolution transformation such as a orthogonal wavelet transformation to the scene-referred raw data d 2  inputted from the header information analysis section  102  to divide them into pertinent frequency ranges, applies a smoothing processing to them by the use of a frequency range reducing filter, and outputs them to the scene-referred image data generator  104 . The smoothing processing is carried out by the use of a filter which changes the mask sizes, the mask shapes, and the threshold values for the comparison with the difference of pixel values between a pixel of remark and its surrounding pixels on the basis of the characteristic of noises. The smoothing processor  113  has the function as a smoothing processing means of an image processing apparatus as set forth in the claims of this invention.  
     [0282] The scene-referred image data generator  104  applies a correction processing for image-capturing device characteristics to the scene-referred raw data d 2  inputted from the smoothing processor  113  on the basis of the generation condition determined by the correction processor for apparatus characteristics  103   a , to generate standardized scene-referred image data which are independent of image-capturing device characteristics, and outputs them to the temporary storage  105 . To state it concretely, in the correction processing for image-capturing device characteristics, at least a processing for mapping signal strength data of each of the color channels based on the spectral sensitivity specific to the image sensor of the image-capturing device having generated the scene-referred raw data d 2  to a standard color space such as the aforesaid RIMM RGB or ERIMM RGB is included. The temporary storage  105  is made up of a buffer storage etc., and temporarily stores scene-referred image data d 4  generated by the scene-referred image data generator  104 .  
     [0283] As shown in FIG. 16, the optimization processor  116  has a structure equipped with a photographing information data processor  106 , a viewing image referred image data generator  107 , a temporary storage  108 , a setting input section  109 , and a sharpening processor  114 . The optimization processor  116  has the function as a viewing image referred image data generation means of an image processing apparatus as set forth in the claims of this invention.  
     [0284] On the basis of the photographing information data d 3  inputted from the header information processor  102 , the photographing information data processor  106  determines generation conditions for generating viewing image referred image data d 5  in accordance with the photographing conditions.  
     [0285] The setting input section  109 , having operation information inputted concerning the kind and size of an output medium, to state it concretely, the kind of the storage device  110 , the output device  111 , and the display device  112 , or the size of the output medium for the output of digital image data generated by this image processing apparatus  117 , outputs this operation information to the viewing image referred image data generator  107 .  
     [0286] When scene-referred image data d 4  are read out from the temporary storage  105  and outputted to the sharpening processor  114 , and a sharpening processing is applied to the scene-referred image data d 4  by the sharpening processor  114 , on the basis of the generation conditions of viewing image referred image data d 5  determined for the image data by the photographing information data processor  106  and the operation information inputted from the setting input section  109 , the viewing image referred image data generator  107  applies an optimization processing for obtaining an optimum image at the output end to the image data, to output the image data to the temporary storage  108  together with the operation information. In the optimization processing, for example, compression to the color gamut of the output medium, gradation compression from 16 bits to 8 bits, reduction of the output number of pixels, a processing to cope with the output characteristics (LUT) of the output device or the display device. Further, image processings such as a noise suppression processing, a color balance adjustment, a chroma adjustment, a dodging processing are also included.  
     [0287] The sharpening processor  114  applies edge enhancement processing to the inputted scene-referred image data d 4  by an enhancement processing using Laplacian or a processing using an unsharp mask. As regards the amount of edge enhancement, by an adjustment in accordance with the kind and size of the output medium designated by the setting input section  109 , a sharpening processing suitable to the output medium is practiced. Further, the sharpening processor  114  discriminates the size of the main photographic object in a viewing image, and adjusts the amount of edge enhancement in accordance with this size. Further, it discriminates the photographed scene in digital image data, and adjusts the amount of edge enhancement in accordance with the photographed scene. The sharpening processor  114  has the function as a sharpening processing means of an image processing apparatus as set forth in the claims of this invention.  
     [0288] The temporary storage  108  is a buffer storage or the like, and stores temporarily viewing image referred image data inputted from the viewing image referred image data generator  107 .  
     Operation of Image Processing Apparatus  117   
     [0289]FIG. 17 is a flow chart showing an image data generation processing A to be practiced by the cooperation of the pertinent portions of the image processing apparatus  117 . In the following, the operation of the image processing apparatus  117  will be explained with reference to the drawings.  
     [0290] When a recording medium having the data structure shown in FIG. 9 is loaded, the digital image data file recorded on the recording medium is inputted (step S 21 ). As regards the inputted digital image data, its content is analyzed by the header information analysis section  102  (step S 22 ), and is divided into scene-referred raw data d 2  (step S 23 ), correction data for image-capturing device characteristics d 1  (step S 24 ), and photographing information data d 3  (step S 25 ); the scene-referred raw data d 2  and the correction data for image-capturing device characteristics d 1  are outputted to the correction processor for image-capturing device characteristics  115 , and the photographing information data are outputted to the optimization processor  116 .  
     [0291] When the correction data for image-capturing device characteristics d 1  are inputted to the correction processor for image-capturing device characteristics  115 , the correction processor for apparatus characteristics  103   a  refers to the processing condition table  103   b , and determines processing conditions for generating scene-referred image data d 4 . The scene-referred raw data d 2  are subjected to a smoothing processing by the smoothing processor  113 , on the basis of the processing conditions determined by the correction processor for apparatus characteristics  103   a  (step S 26 ), are subjected to a correction processing for image-capturing device characteristics by the scene-referred image data generator  104  (step S 27 ), and becomes scene-referred image data generated, which are stored in the temporary storage  105  (step S 28 ).  
     [0292] When the photographing information data d 3  are inputted to the optimization processor  116 , processing conditions for generating viewing image referred image data in accordance with the photographing conditions are determined by the photographing information data processor  106  on the basis of the photographing information data d 3 . The scene-referred image data d 4  are read out from the temporary storage  105  of the correction processor for image-capturing device characteristics  115 , are subjected to a sharpening processing by the sharpening processor  114 , on the basis of the processing conditions determined by the photographing information data processor  106  and the operation information inputted from the setting input section  109  (step S 29 ), and are subjected to an optimization processing by the viewing image referred image data generator  107  (step S 30 ), to become viewing image referred image data generated, which are outputted to the device designated by the setting input section  109  (step S 31 ).  
     [0293] In addition, the block diagram shown in FIG. 16 and the flow chart shown in FIG. 17 are merely an example; as regards the processing by the smoothing processor  113 , it is appropriate so long as it is practiced in the process of generating scene-referred image data d 4  from scene-referred raw data d 2 , and as regards the processing by the sharpening processor  114 , it is appropriate so long as it is practiced in the process of generating viewing image referred image data d 5  from scene-referred image data d 4 ; the process is not limited to the block diagram shown in FIG. 16 and the flow chart shown in FIG. 17. For example, it is also appropriate that scene-referred raw data d 2  are outputted from the header information processor  102  to the scene-referred image data generator  104 , and after being subjected to several correction processings for image-capturing device characteristics, are subjected to a smoothing processing in the smoothing processor  113 . Further, it is also appropriate that scene-referred image data are read out from the temporary storage  105  by the viewing image referred image data generator  107 , and after being subjected to several optimization processings, are subjected to a sharpening processing in the sharpening processor  114 .  
     [0294] As explained in the foregoing, by the image processing apparatus  117 , a smoothing processing is applied to digital image data inputted from the image-capturing device  21  in the process of generating scene-referred image data d 4  from scene-referred raw data d 2 . The generation process of scene-referred image data d 4  is a process to correct captured-image data for image-capturing device characteristics, and it is in a state such that the information volume (for example, the number of gray levels and the number of pixels) of the digital image data to become the object of processing is large, and the cause of noise generation, which is specific to each image-capturing device, is clearer; therefore, a more suitable smoothing processing can be practiced.  
     [0295] Further, by the image processing apparatus  117 , a sharpening processing is applied to scene-referred image data d 4  in the generation process of viewing image referred image data d 5 . The generation process of viewing image referred image data d 5  is a process to practice image processings for optimizing digital image data in accordance with the output device or the output medium, and by the application of a sharpening processing in accordance with the output device or the output medium, a subjectively desirable image can be obtained on the output medium.  
     Structure of Image Recording Apparatus  201   
     [0296] In the following, a desirable mode of practice of an image recording apparatus of this invention will be explained.  
     [0297] In addition, an image recording apparatus  201  in this embodiment 1 corresponds to an image recording apparatus as set forth in the structure (11) of this invention, the combination of a correction processor for image-capturing device characteristics  303   a , a scene-referred image data generator  304 , and a smoothing processor  313  has the function as a scene-referred image data generation means as set forth in the structure (11) of this invention, and the combination of a photographing information data processor  306 , a viewing image referred image data generator  307 , and a sharpening processor  314  has the function as a viewing image referred image data generation means as set forth in the structure (11) of this invention. Further, the smoothing processor  313  has the function as a smoothing processing means as set forth in the structure (11) of this invention, and the sharpening processor  314  has the function as a sharpening processing means as set forth in the structure (11) of this invention.  
     [0298]FIG. 18 is a perspective view showing the outside structure of the image recording apparatus  201  of this invention. The image recording apparatus in this mode of practice is an example of the apparatus which is equipped with a CRT display monitor as a display device, and an output device using a silver halide photographic paper as an output medium.  
     [0299] In the image recording apparatus  201 , there are provided a magazine loading section  203  at the left side of the mainframe  202 , an exposure processor  204  for exposing to light a silver halide photographic paper as an output medium inside the mainframe  202 , and a print producing section  205  for producing a print through the application of a development processing to the exposed silver halide photographic paper and drying it. The produced print is ejected to a tray  206  provided at the right side of the mainframe  202 . Further, inside the mainframe  202 , a controller  207  is provided at a position above the exposure processor  204 .  
     [0300] Further, on the upper surface of the mainframe  202 , a CRT  208  is disposed. This CRT  208  has the function of a display means for displaying the image of the image information for which a print is to be produced. At the left side of the CRT  208 , a film scanner section  209 , which is a transmission original reading device, is disposed, and at the right side of the CRT  208 , a reflection copy input device  210  is disposed.  
     [0301] For an original to be read by the film scanner section  209  or the reflection copy input device  210 , a photographic photosensitive material can be cited. For this photographic photosensitive material, a color negative film, a color reversal film, a black-and-white negative film, a black-and-white reversal film, etc. can be cited, and frame image information obtained by the photographing of an object by means of an analog camera is recorded on them. A film scanner in the film scanner section  209  can convert this recorded frame image information into digital image data, to generate frame image data. Further, in the case where the photographic photosensitive material is a color paper, which is a silver halide photographic paper, the frame image information is converted into frame image data by the flat bed scanner of the reflection copy input device  210 .  
     [0302] At the position where the controller  207  is disposed in the mainframe  202 , there is provided an image data reading section  214 . The image data reading section  214  is equipped with an adapter for a PC card  214   a  and an adapter for an FD (floppy (registered trade mark) disk)  214   b , and it is possible to insert a PC card  213   a  and an FD  213   b  into them respectively. The PC card  213   a  has a storage having image data of a plurality of image frames obtained by the image pickup using a digital camera stored. The FD  213   b  has image data of a plurality of image frames obtained by the image pickup using, for example, a digital camera stored.  
     [0303] At this side of the CRT  208 , an operation section  211  is disposed, and this operation section  211  is equipped with an information input means  212 . The information input means  212  consists, for example, of a touch panel, or the like.  
     [0304] For a recording medium having frame image data of this invention other than the above-mentioned, a multi-media card, a memory stick, an MD data, a CD-ROM, etc. can be cited. In addition, the apparatus has a structure that the operation section  211 , the CRT  208 , the film scanner section  209 , the reflection copy input device  210 , and the image data reading section  214  are integrally built in the mainframe  202 , but it is also appropriate to provide any one or more of them as a separate body.  
     [0305] Further, at the position where the controller  207  is disposed in the mainframe  202 , an image data writing section  215  is provided. The image data writing section  215  is equipped with an adapter for an FD  215   a , an adapter for an MO  215   b , and an adapter for an optical disk  215   c ; it is possible to insert an FD  216   a , an MO  216   b , and an optical disk  216   c  into them respectively, to make it possible to write image information in an image recording medium.  
     [0306] Further, the controller  207  is equipped with a communication means (not shown in the drawing), which makes it possible for the controller to receive image data representing pickup image and a print instruction directly from another computer in the same facilities or a computer located at a remote site connected through a network, to make the image recording apparatus function as what is called a network image output apparatus.  
     Internal Structure of Image Recording Apparatus  201   
     [0307] Next, the internal structure of the image recording apparatus  201  will be explained.  
     [0308]FIG. 19 is a block diagram showing the internal structure of the image recording apparatus  201 .  
     [0309] The controller  207  of the image recording apparatus  201  is composed of a CPU (Central Processing Unit), a storage section, etc. The CPU reads out various kinds of control program stored in the storage section, and executes a concentrated control of the pertinent parts making up the image recording apparatus in accordance with said control programs read out.  
     [0310] Further, the controller  207  comprises an image processor  270 , and applies image processing, on the basis of an input signal from the information input means  212  of the operation section  211 , to image data obtained by making the film scanner section  209  or the reflection copy input section  210  read an original image, to image data read out from the image data reading section  214 , and to image data inputted from external equipment through a communication means (input)  240  (shown in FIG. 20). Further, in the image processor  270 , a transformation processing in accordance with the mode of output is applied to the image data having been subjected to the image processing, and the image data are outputted as a print P 1 , P 2 , or P 3 , or by means of the CRT monitor  208 , image data writing section  215 , the communication means (output)  241 , or the like.  
     [0311] In the operation section  211 , there is provided the information input means  212 . The information input means  212  is made up, for example, of a touch panes or the like, and outputs its depression signal as an input signal to the controller  207 . Further, the operation section  211  may have a structure equipped with a keyboard and a mouse.  
     [0312] The film scanner section  209  reads a frame image of a developed negative film N obtained by the developing of a negative film subjected to photographing by an analog camera, to generate frame image data, and the reflection copy input device  210  reads a frame image of a print P obtained by the exposing of a color paper as a silver halide photographic paper to the frame image and the application of a development processing to it, to generate frame image data.  
     [0313] The image data reading section  214  has the function such that it reads out and transfers frame image data of the PC card  213   a  or the FD  213   b  which have been obtained through the photographing by means of an analog camera and stored. That is, the image data reading section  214  is equipped with an adapter for a PC card, an adapter for an FD, etc. as an image transfer means  230 , and reads out frame image data recorded in the PC card  213   a  loaded in the adapter for a PC card  214   a  or in the FD  213   b  loaded in the adapter for an FD  214   b , and transfers them to the controller  207 . For the adapter for a PC card  214   a , for example, a PC card reader, a PC card slot, or the like is used.  
     [0314] A data accumulation means  271  stores image information and order information (information on how many prints are to be produced from image data of which frame, information on print size, etc.) corresponding to the image information, and accumulates them successively.  
     [0315] A template storage means  272  stores sample image data (data indicating a background image, an illustration image, or the like) and the data of at least one template for defining a composite area with said sample image data, in correspondence to sample identification information D 1 , D 2 , and D 3 . Here, when a specified template is selected out of a plurality of templates stored beforehand in the template storage means  272  by the operation of an operator (this operation of an operator is based on the instruction of a client), the controller  207  combines frame image information and said selected template, subsequently, when any one of three kinds of sample identification information D 1 , D 2 , and D 3  is designated by the operation of the operator (this operation of an operator is based on the instruction of a client), the controller selects sample image data on the basis of said selected any one of three kinds of sample identification information D 1 , D 2 , and D 3 , and combines said selected sample image data with the image data and/or character data based on the order of the client, to produce a print based on the sample image data desired by the client as the result. This composition processing using a template is carried out by a well-known chroma key method.  
     [0316] In addition, as regards the sample identification information, the kinds of it is not limited to the three kinds D 1 , D 2 , and D 3 , but they may be more than three or less than three.  
     [0317] Further, in the above-mentioned description, the three kinds of sample identification information D 1 , D 2 , and D 3  for designating the sample of a print are to be inputted from the operation section  211 , but because the three kinds of sample identification information D 1 , D 2 , and D 3  are recorded in a print sample, or in an order sheet, they can be read by a reading means such as an OCR. In another way, they can be inputted from a keyboard by an operator.  
     [0318] In this way, sample image data are recorded beforehand in correspondence to sample identification D 1  designating a sample of a print, the sample identification information D 1  to designate a sample of a print is inputted, a sample image data are selected on the basis of this inputted sample identification information D 1 , and the selected sample image data and image data and/or character data based on the order are combined together to become composite data, from which a print based on the designated sample is produced; therefore, it is possible for a user to make an order for a print with various kinds of life-size sample taken in hands, and this method makes it possible to respond diversified demands of wide-ranging users.  
     [0319] Further, the first-kind sample identification information D 2  for designating a first sample and image data of the first sample are stored, and the second-kind sample identification information D 3  for designating a second sample and image data of the second sample are stored, sample image data selected on the basis of the first-kind and the second-kind sample identification information D 2  and D 3  to be designated are combined with image data and/or character data based on an order to become composite data, from which a print based on the designated sample is produced; therefore, it is possible to compose more diversified images of more kinds, and it is possible to produce a print responding to diversified demands of wider-ranging users.  
     [0320] The exposure processor  204  applies an exposure processing to a photosensitive material in accordance with image data for output which are generated from image data subjected to image processing in the image processor  270 , and sends this photosensitive material to the print producing section  205 . The print producing section  205  applies a development processing to the exposed photosensitive material and dries it, to produce a print P 1 , P 2 , or P 3 . The print P 1  is of the service size, of the high-vision size, of the panorama size, or the like, the print P 2  is of the A4 size, and the print P 3  is of the calling-card size.  
     [0321] In addition, the print sizes are not limited to the prints P 1 , P 2 , and P 3 , but any other size may be possible.  
     [0322] The monitor  208  is made up of a CRT, an LCD, or the like, and displays image information inputted from the controller  207 .  
     [0323] The image data writing section  215  is equipped with an adapter for an FD  215   a , an adapter for an MO  215   b , and an adapter for an optical disk  216   c  as an image transport section  231 ; it is possible to insert an FD  216   a , an MO  216   b , and an optical disk  216   c  respectively into them, and the image data writing section  215  makes it possible to write image data in an image recording medium.  
     [0324] Further, it is also possible that the image processor  270 , using the communication means (input)  240  (shown in FIG. 20), receives image data representing a pickup image and an operation instruction such as printing directly from another computer in the same facilities or a computer located at a remote site connected through the Internet or the like, and practices image processing or produces a print by a remote-control operation.  
     [0325] Further, it is also possible that the image processor  270 , using a communication means (output)  241  (shown in FIG. 20), transmits image data after being subjected to image processing of this invention and incidental order information to another computer in the same facilities or to a computer located at a remote site connected through the Internet or the like.  
     [0326] As explained in the above, the image recording apparatus  201  comprises an input means for taking image information of various kinds of digital medium and image information obtained by reading an original image, an image processing means for practicing processings so as to produce an image giving a good impression to a person observing the image on an output medium by obtaining or estimating the information on the “size of the viewing image” and the “size of the main photographic object in the viewing image” from the image information of an input image taken from this input means, an image output means for displaying, outputting as a print, or writing in an image recording medium the processed image data, and a communication means (transmission) for transmitting image data and incidental order information to another computer in the same facilities connected through a communication network or a computer located at a remote site connected through the Internet or the like.  
     Structure of Image Processor  270   
     [0327]FIG. 20 is a block diagram showing the functional structure of the image processor  270  of this invention. Image data inputted from the film scanner section  209  are subjected to a correction operation specific to the film scanner section, negative-to-positive reversing in the case of a negative original, removal of dusts and scratches, a gray balance adjustment, a contrast adjustment, removal of a granular noise, a sharpness enhancement, etc. in a film scanner data processor  702 , and are sent to an image adjustment processor  701 . Further, information on film size, information concerning whether a film is negative or positive, information concerning the main photographic object recorded optically or magnetically on a film, information concerning the photographing conditions (for example, the content of noted information in APS (Advanced Photo System)), etc. are outputted together to the image adjustment processor  701 .  
     [0328] Image data inputted from the reflection copy input device  210  are subjected, in a reflection copy scan data processor  703 , to a correction operation specific to the reflection copy input device  210 , a negative-to-positive reversing processing in the case of a negative original, removal of dusts and scratches, a gray balance adjustment, a contrast adjustment, removal of a noise, a sharpness enhancement, etc., and are outputted to the image adjustment processor  701 .  
     [0329] Image data inputted from the image transfer means  230  or the communication means (input)  240  are subjected, as occasion demands, to decoding of compressed codes, transformation of color data representation system, etc. in accordance with the data format of the data, being transformed into a data format suitable to the operation in the image processor  270 , and are outputted to the image adjustment processor  701 . Further, a decoding processor for image data format  704  judges whether or not image data of a format depending on the above-mentioned image-capturing device are inputted from the image transfer means  230  or the communication means (input)  240 , and if it judges that image data are inputted from the image transfer means  230  or the communication means (input)  240 , it outputs the inputted image data to a header information analysis section  302 . In the header information analysis section  302 , input image data are analyzed, and divided into correction data for image-capturing device characteristics d 1 , scene-referred raw data d 2 , and photographing information data d 3 .  
     [0330] The designation of the size of a viewing image is inputted from the operation section  211 , and in the case where, in addition to the above-mentioned, there is a designation about the size of a viewing image transmitted to the communication means (input)  240 , or a designation of the size of a viewing image which is buried in the header information/tag information of image data obtained by the image transfer means  230 , the decoding processor for image data format  704  detects said information, and transfers it to the image adjustment processor  701 .  
     [0331] Correction data for image-capturing device characteristics d 1  analyzed by the header information analysis section  302  are outputted to a correction processor for apparatus characteristics  303   a , and image processing conditions are determined on the basis of a processing condition table  303   b.    
     [0332] The smoothing processor  313  applies, for example, an orthogonal transformation such as a DCT (Discrete Cosine Transformation), a Fourier transformation, or a multi-resolution transformation such as an orthogonal wavelet transformation to the scene-referred raw data d 2  inputted from the header information analysis section  302  to divide them into pertinent frequency ranges, practices the smoothing of an image by the use of a low-pass filter of frequency ranges, and outputs them to a scene-referred image data generator  304 . The smoothing processing is carried out by the use of a filter which changes the mask sizes, the mask shapes, and the threshold values for the comparison with the difference of pixel values between a pixel of remark and its surrounding pixels on the basis of the characteristic of noises.  
     [0333] The scene-referred image data generator  304  applies a correction processing for image-capturing device characteristics to the scene-referred raw data d 2  inputted from the smoothing processor  313  on the basis of the generation condition determined by the correction processor for apparatus characteristics  303   a , to generate standardized scene-referred image data which are independent of image-capturing device characteristics, and outputs them to a viewing image referred image data generator  307 . To state it concretely, in the correction processing for image-capturing device characteristics, at least a processing for mapping signal strength data of each of the color channels based on the spectral sensitivity specific to the image sensor of the image-capturing device having generated the scene-referred raw data d 2  to a standard color space such as the aforesaid RIMM RGB or ERIMM RGB is included.  
     [0334] The photographing image data analyzed by the header information analysis section  302  are outputted to the photographing information data processor  306 , and image processing conditions concerning the generation of viewing image referred image data are determined.  
     [0335] On the basis of an instruction from the operation section  211  and the controller  207 , the image adjustment processor  701  transfers image processing conditions for generating viewing image referred image data d 5  adapted to the output medium to the viewing image referred image data generator  307 .  
     [0336] A sharpening processor  314  practices an edge enhancement processing for inputted scene-referred image data d 4  by an enhancement processing using Laplacian or a processing using an unsharp mask. As regards the amount of edge enhancement, by the adjusting of it in accordance with the kind and size of the output medium designated by the operation section  211 , a sharpening processing suitable to the output medium is practiced. Further, the sharpening processor  314  discriminates the size of the main photographic object in a viewing image, and adjusts the amount of edge enhancement in accordance with this size. Further, it discriminates the photographed scene in a viewing image, and adjusts the amount of edge enhancement in accordance with the photographed scene.  
     [0337] On the basis of the image processing conditions produced by the photographing information data processor  306  and the image processing conditions transmitted from the image adjustment processor  701 , the viewing image referred image data generator  307  applies an optimization processing to the inputted scene-referred image data d 4  to generate viewing image referred image data d 5 , and outputs the image data to the image adjustment processor  701  together with the operation information. In the optimization processing, for example, compression to the color gamut of the output medium, gradation compression from 16 bits to 8 bits, reduction of the output number of pixels, a processing to cope with the output characteristics (LUT) of the output device or the display device. Further, image processings such as a noise suppression processing, a color balance adjustment, a chroma adjustment, a dodging processing are also included.  
     [0338] The image adjustment processor  701  calls out specified image data (template) from the template storage means  272  if a template processing is required. It transfers image data to a template processor  705 , where the data are combined with a template, and receives the image data after the template processing again. Further, the image adjustment processor  701 , on the basis of an instruction of the operation section  211  or the controller  207 , applies image processing so as to produce an image giving a good impression to a person observing the image on the output medium, to image data received from the film scanner section  209 , the reflection copy input device  210 , the image transfer means  230 , the communication means (input)  240 , and the template processor  705 , to generate digital image data for output, and sends out the image data to a CRT-specific processor  706 , a printer-specific processor (1) 707, an image data format generating processor  709 , or the data accumulating means  271 .  
     [0339] The CRT-specific processor  706  applies processings such as changing of the number of pixels and color matching to image data received from the image adjustment processor  701 , and sends out composite image data for display obtained by combining the received image data with information requiring display such as control information to the CRT  208 . The printer-specific processor  707  practices a correction processing as occasion demands, color matching, changing of the number of pixels, etc., and sends out image data to the exposure processor. In the case where an external printer  251  such as an ink jet printer of a large print size is further connected to the image recording apparatus  201  of this invention, it is practiced that a printer-specific processor (2)  708  is provided for each printer connected, and a proper correction processing specific to the printer, color matching, changing of the number of pixels, etc. are carried out.  
     [0340] The image data format generating processor  709  applies compression of data and transformation into various kinds of versatile image format represented by JPEG, TIFF, Exif, etc. as occasion demands to image data received from the image adjustment processor  701 , and transfers the image data to the image transport section  231  or the communication means (output)  241 .  
     [0341] Besides, viewing image referred image data d 5  generated in the viewing image referred image data generator  307  are such that are premised on the processing in the CRT-specific processor  706 , the printer-specific processor (1)  707 , the printer-specific processor (2)  708 , and the image data format generation processor  709  described in the above, and it is possible that, in the image data format generation processor  709 , on the basis of the format of viewing image referred image data, a status file indicating that the data are optimized image data for the CRT, the exposure output section, an external printer, a communication means (output), or the like is attached to the image data, and the image data optimized for each output end with each status file attached are transmitted to the image transport section separately and are saved.  
     [0342] The above-mentioned sections, the film scan data processor  702 , the reflection copy scan data processor  703 , decoding processor for image data format  704 , the image adjustment processor  701 , the CRT-specific processor  706 , the printer-specific processor (1)  707 , the printer-specific processor (2)  708 , and the image data format generation processor  709  are the sections provided for the purpose of assisting the comprehension of the function of the image processor  270 , and it is unnecessary that they are always actualized as physically independent devices; it is also appropriate to actualize them, for example, as the sections of the kinds of software processing in a single CPU.  
     [0343] Further, the sections, the header information analysis section  302 , the correction processor for apparatus characteristics  303   a , the photographing information data processor  306 , the scene-referred image data generator  304 , the viewing image referred image data generator  307 , the smoothing processor  313 , and the sharpening processor  314  are the sections provided for the purpose of assisting the comprehension of the function of the image processor  270 , and it is unnecessary that they are always actualized as physically independent devices; it is also appropriate to actualize them, for example, as the sections of the kinds of soft ware processing in a single CPU.  
     Operation of Image Processor  270   
     [0344]FIG. 21 is a flow chart showing an image data formation processing A to be practiced by the cooperation of the pertinent portions of the image processor  270 . In the following, the operation of every part of the image processor will be explained with reference to the drawings.  
     [0345] When data are inputted from the image transfer means  230  or the communication means (input)  240  to the image processor  270  (step S 41 ), and said inputted data are judged by the decoding processor for image data format  704  as a digital image data file obtained by the above-mentioned image-capturing device  21  (step S 42 ), the content of the inputted digital image data file is analyzed by the header information analysis section  302  (step S 43 ), and is divided into scene-referred raw data d 2  (step S 44 ), correction data for image-capturing device characteristics d 1  (step S 45 ), and photographing information data d 3  (step S 46 ).  
     [0346] The correction data for image-capturing device characteristics d 1  are outputted to the correction processor for apparatus characteristics  303   a , and by the correction processor for apparatus characteristics  303   a  referring to the processing condition table  303   b , processing conditions for generating scene-referred image data d 4  are determined. The scene-referred raw data d 2  are subjected to a smoothing processing by the smoothing processor  313  on the basis of the processing conditions determined by the correction processor for apparatus characteristics  303   a  (step S 47 ), and are subjected to a correction processing for image-capturing device characteristics by the scene-referred image data generator  304  (step S 48 ), to become scene-referred image data d 4  generated, which are outputted to the viewing image referred image data generator  307  (step S 49 ).  
     [0347] When the photographing information data d 3  are inputted to the photographing information data processor  306 , processing conditions for generating viewing image referred image data d 5  in accordance with the photographing conditions are determined on the basis of the photographing information data d 3 . Further, on the basis of an instruction from the operation section  211  and the controller  207 , in the image adjustment processor  701 , image processing conditions for generating viewing image referred image data d 5  adapted to the kind and size of the device or the output medium of the output end are determined. The scene-referred image data d 4  are subjected to a sharpening processing by the sharpening processor  314  on the basis of the processing conditions determined by the photographing information data processor  306  and the image processing conditions determined by the image adjustment processor  701  (step S 50 ), and are subjected to an optimization processing by the viewing image referred image data generator  307  (step S 51 ). Then, viewing image referred image data d 5  are generated, and are outputted, in accordance with the output end, to any one of the processors, the CRT-specific processor  706 , the printer-specific processor  707 , the printer-specific processor  708 , and the image data format generation processor  709  (step S 52 ). The viewing image referred image data d 5  are subjected to processings specific to the output end in accordance with it (step S 53 ), and are outputted to the output end designated by the operation section  211  (step S 54 ).  
     [0348] In addition, the flow chart shown in FIG. 21 is merely an example; as regards the processing by the smoothing processor  313 , it is appropriate so long as it is practiced in the process of generating scene-referred image data d 4  from scene-referred raw data d 2 , and as regards the processing by the sharpening processor  314 , it is appropriate so long as it is practiced in the process of generating viewing image referred image data d 5  from scene-referred image data d 4 ; the process is not limited to the flow chart shown in FIG. 21. For example, it is also appropriate that scene-referred raw data d 2  are outputted from the header information processor  302  to the scene-referred image data generator  304 , and after being subjected to several correction processings for image-capturing device characteristics in the scene-referred image data generator  304 , are subjected to a smoothing processing in the smoothing processor  313 . Further, it is also appropriate that scene-referred image data, after being subjected to several optimization processings by the viewing image referred image data generator  307 , are subjected to a sharpening processing in the sharpening processor  314 .  
     [0349] As explained in the foregoing, by the image recording apparatus  201 , a smoothing processing is applied to digital image data inputted from the image-capturing device  21  in the process of generating scene-referred image data d 4  from scene-referred raw data d 2 . The generation process of scene-referred image data d 4  is a process to correct captured-image data for image-capturing device characteristics, and it is in a state such that the information volume (for example, the number of gray levels and the number of pixels) of digital image data to become the object of processing is large, and the cause of noise generation, which is specific to each image-capturing device, is clearer; therefore, a more suitable smoothing processing can be practiced.  
     [0350] Further, by the image recording apparatus  201 , a sharpening processing is applied to digital image data inputted from the image-capturing device  21  in the generation process of viewing image referred image data d 5 . The generation process of viewing image referred image data d 5  is a process to practice image processing for optimizing digital image data in accordance with the output medium, and by the application of a sharpening processing in accordance with the output medium, a subjectively desirable image can be obtained on the output medium.  
     [0351] The image processing apparatus  117  and the image recording apparatus  201  of this embodiment 1 apply a smoothing processing to input data having comparatively large noises for the purpose of reducing the processing time. In this embodiment 1, a smoothing processor is provided in the image processing apparatus, but a smoothing processor can be provided in the image-capturing device. Because a sharpening processing is practiced in the process of generating viewing image referred image data in this embodiment 1, a sharpening processing is practiced before data compression is practiced. It has been found that, for this reason, a sharpening processing which is excellent in the sense of total balance with the color gamut and the gradation of the image can be practiced and a good image can be obtained on any one of output media of various kinds.  
     [0352] Embodiment 2  
     [0353] In the following, the embodiment 2 of this invention will be explained with reference to the drawings.  
     [0354] In addition, in the embodiment 2, in the same manner as explained in the above-mentioned embodiment 1, it is shown an example using digital image data recorded by the image-capturing device  21 .  
     Structure of Image Processing Apparatus  117   
     [0355]FIG. 22 is a block diagram showing the functional structure of an image processing apparatus  117  of this invention. As shown in FIG. 22, the arrangement of a sharpening processor  114  is different from that in the image processing apparatus  117  of the embodiment 1 shown in FIG. 16. That is, scene-referred image data d 4  stored in a temporary storage  105  are read out by a viewing image referred image data generator  107  of an optimization processor  116 , and are subjected to an optimization processing in accordance with the output end to become viewing image referred image data d 5  generated. After that, the viewing image referred image data d 5  are subjected to a sharpening processing by the sharpening processor  114 , and sharpened viewing image referred image data d 6  are generated.  
     [0356] The other parts of the structure of the image processing apparatus  117  of this embodiment 2 are the same as those of the above-mentioned embodiment 1; therefore, every component is given the same sign as that of the embodiment 1, and the explanation of the structure will be omitted.  
     Operation of Image Processing Apparatus  117   
     [0357]FIG. 23 is a flow chart showing an image data generation processing B to be practiced by the cooperation of the pertinent portions of the image processing apparatus  117 . In the following, the operation of the image processing apparatus  117  will be explained with reference to the drawings.  
     [0358] When a recording medium having recorded a data file of the structure shown in FIG. 9 is loaded, the digital image data file recorded in the recording medium is inputted by the input section  101  (step S 61 ). As regards the inputted digital image data, its content is analyzed by the header information analysis section  102  (step S 62 ), and is divided into scene-referred raw data d 2  (step S 63 ), correction data for image-capturing device characteristics d 1  (step S 64 ), and photographing information data d 3  (step S 65 ); the scene-referred raw data d 2  and the correction data for image-capturing device characteristics d 1  are outputted to the correction processor for image-capturing device characteristics  115 , and the photographing information data d 3  are outputted to the optimization processor  116 .  
     [0359] When the correction data for image-capturing device characteristics d 1  are inputted to the correction processor for image-capturing device characteristics  115 , the processing condition table  103   b  is referred to by the correction processor for apparatus characteristics  103   a , and processing conditions for generating scene-referred image data d 4  are determined. The scene-referred raw data d 2  are subjected to a smoothing processing by the smoothing processor  113  on the basis of the processing conditions determined by the correction processor for apparatus characteristics  103   a  (step S 66 ), are subjected to a correction processing for image-capturing device characteristics by the scene-referred image data generator  104  (step S 67 ), to become scene-referred image data d 4  generated, which are stored in the temporary storage  105  (step S 68 ).  
     [0360] When the photographing information data d 3  are inputted to the optimization processor  116 , processing conditions for generating viewing image referred image data d 5  in accordance with the photographing conditions are determined by the photographing information data processor  106  on the basis of the photographing information data d 3 . Scene-referred image data d 4  are read out from the temporary storage  105 , and on the basis of the processing conditions determined by the photographing information data processor  106  and the operation information inputted from the setting input section  109 , are subjected to an optimization processing in accordance with the size of the device or the output medium of the output end (step S 69 ), to become viewing image referred image data d 5  generated (step S 70 ). The generated viewing image referred image data d 5  are subjected to a sharpening processing in the sharpening processor  114  (step S 71 ), to become sharpened viewing image referred image data d 6  generated, which are outputted to the device designated by the setting input section  109  (step S 72 ).  
     [0361] In addition, the block diagram shown in FIG. 22 and the flow chart shown in FIG. 23 are merely an example; as regards the processing by the smoothing processor  113 , it is appropriate so long as it is practiced in the process of generating scene-referred image data d 4  from scene-referred raw data d 2 ; the process is not limited to the block diagram shown in FIG. 22 and the flow chart shown in FIG. 23. For example, it is also appropriate that scene-referred raw data d 2  are outputted from the header information processor  102  to the scene-referred image data generator  104 , and after being subjected to several correction processings for image-capturing device characteristics, are subjected to a smoothing processing in the smoothing processor  113 .  
     [0362] As explained in the foregoing, by the image processing apparatus  117 , a smoothing processing is applied to digital image data inputted from the image-capturing device  21  in the process of generating scene-referred image data d 4  from scene-referred raw data d 2 . The generation process of scene-referred image data d 4  is a process to correct captured-image data for image-capturing device characteristics, and it is in a state such that the information volume (for example, the number of gray levels and the number of pixels) of digital image data to become the object of processing is large, and the cause of noise generation, which is specific to each image-capturing device, is clearer; therefore, a more suitable smoothing processing can be practiced.  
     [0363] Further, by the image processing apparatus  117 , a sharpening processing is applied to scene-referred image data d 4  after the generation process of viewing image referred image data d 5 . For that reason, in a case such that viewing image referred image data d 5  generated through the application of an optimization processing based on the specified kind of the output device or the output medium are outputted to the output device or the output medium of the same kind with the size of the output media changed, an optimization processing becomes unnecessary; therefore, the processing load can be lightened.  
     Structure of Image Recording Apparatus  201   
     [0364] Next, the structure and the operation of an image recording apparatus  201  of the embodiment 2 of this invention will be explained. In addition, because the outside structure and the internal structure of the image recording apparatus  201  and the structure of an image processor  270  of this embodiment 2 are the same as those of the above-mentioned embodiment 1 except that the order of data processing is different, the feature is to be explained in the explanation of the operation; further, every structural component is given the same sign as that of the embodiment 1, and it will be omitted to show the structure in the drawings and the explain it. Further, the image recording apparatus  201  of this embodiment 2 is one corresponding to an image recording apparatus as set forth in the structure (12) of this invention, the combination of the correction processor for image-capturing device characteristics  303   a , the scene-referred image data generator  304 , and the smoothing processor  313  together has the function of a scene-referred image data generation means as set forth in the structure (12) of this invention, and the combination of the photographing information processor  306 , the viewing image referred image data processor  307 , and the sharpening processor  314  together has the function of a viewing image referred image data generation means as set forth in the structure (12) of this invention. Further, the smoothing processor  313  has the function of a smoothing processor as set forth in the structure (12) of this invention, and the sharpening processor  314  has the function of a sharpening processor as set forth in the structure (12) of this invention.  
     Operation of Image Recording Apparatus  201   
     [0365]FIG. 24 is a flow chart showing an image data formation processing B to be practiced by the cooperation of the pertinent portions of the image processor  270 . In the following, the operation of every part of the image processor  270  will be explained with reference to the drawings.  
     [0366] Data are inputted from the image transfer means  230  or the communication means (input)  240  to the image processor  270  (step S 81 ), and when said inputted data are judged by the decoding processor for image data format  704  as a digital image data file obtained by the aforesaid image-capturing device  21  (step S 82 ), as regards the inputted digital image data file, its content is analyzed by the header information analysis section  302  (step S 83 ), and is divided into scene-referred raw data d 2  (step S 84 ), correction data for image-capturing device characteristics d 1  (step S 85 ), and photographing information data d 3  (step S 86 ).  
     [0367] The correction data for image-capturing device characteristics d 1  are outputted to the correction processor for apparatus characteristics  303   a , and processing conditions for generating scene-referred image data are determined by the correction processor for apparatus characteristics  303   a  referring to the processing condition table  303   b . The scene-referred raw data d 2  are subjected to a smoothing processing by the smoothing processor  313  on the basis of the processing conditions determined by the correction processor for apparatus characteristics  303   a  (step  87 ), and are subjected to a correction processing for image-capturing device characteristics by the scene-referred image data generator  304  (step S 88 ), to become scene-referred image data d 4  generated, which are outputted to the viewing image referred image data generator.  307  (step S 89 ).  
     [0368] When the photographing information data d 3  are inputted to the photographing information data processor  306 , processing conditions for generating viewing image referred image data d 5  in accordance with the photographing conditions are determined on the basis of the photographing information data d 3 . Further, on the basis of an instruction from the operation section  211  and the controller  207 , in the image adjustment processor  701 , image processing conditions for generating viewing image referred image data d 5  adapted to the output device and the output medium are determined. The scene-referred image data d 4  are subjected to an optimization processing by the viewing image referred image data generator  307  in accordance with the size of the output device and the output medium of the output end, on the basis of the processing conditions determined by the photographing information data processor  306  and the image processing conditions determined by the image adjustment processor  701  (step S 90 ), to become viewing image data generated (step S 91 ).  
     [0369] The generated viewing image referred image data d 5  are subjected to a sharpening processing by the sharpening processor  314  (step S 92 ), to become sharpened viewing image referred image data d 6  generated, which are outputted to any one of the processors, the CRT-specific processor  706 , the printer-specific processor  707 , the printer-specific processor  708 , and the image data format generation processor  709  in accordance with the output end based on the operation at the operation section  211  (step S 93 ). The sharpened viewing image referred image data d 6  are subjected to a processing specific to the output end in accordance with it in the processor to which the data are outputted (step S 94 ), and are outputted to the output end designated by the operation section  211  (step S 95 ).  
     [0370] In addition, the flow chart shown in FIG. 24 is merely an example; as regards the processing by the smoothing processor  313 , it is appropriate so long as it is practiced in the process of generating scene-referred image data d 4  from scene-referred raw data d 2 , the process is not limited to the flow chart shown in FIG. 24. For example, it is also appropriate that scene-referred raw data d 2  are outputted to the scene-referred image data generator  304  by the header information processor  302 , and after being subjected to several correction processings for image-capturing device characteristics in the scene-referred image data generator  304 , are subjected to a smoothing processing in the smoothing processor  313 .  
     [0371] As explained in the foregoing, by the image recording apparatus  201 , a smoothing processing is applied to digital image data inputted from the image-capturing device  21  in the process of generating scene-referred image data d 4  from scene-referred raw data d 2 . The generation process of scene-referred image data d 4  is a process to correct captured-image data for image-capturing device characteristics, and it is in a state such that the information volume (for example, the number of gray levels and the number of pixels) of digital image data to become the object of processing is large, and the cause of noise generation, which is specific to each image-capturing device, is clearer; therefore, a more suitable smoothing processing can be practiced.  
     [0372] Further, by the image recording apparatus  201 , a sharpening processing is applied to scene-referred image data d 4  after the generation process of viewing image referred image data d 5 . For that reason, in a case such that viewing image referred image data d 5  generated through the application of an optimization processing based on the specified kind of the output device and the output medium are outputted to the output device and the output medium of the same kind with the size of the output media changed, an optimization processing becomes unnecessary; therefore, the processing load can be lightened.  
     [0373] As described in the above, by the structure of this embodiment 2, scene-referred raw data, scene-referred image data having been subjected to a smoothing processing, and viewing image referred image data having been subjected to a smoothing processing and a sharpening processing can be obtained. As regards the smoothing processing of this embodiment 2, for the purpose of reducing the processing time, it is applied to input data having comparatively large noises. A smoothing processor is provided in the image processing apparatus in this embodiment 2, but a smoothing processor can be also provided in the image-capturing device. Because a sharpening processing is practiced in a process after a viewing image referred image data generation processing, a sharpening processing can be practiced on the basis of output information. It has been found out that, for this reason, an image having a very excellent sharpness can be obtained for a specified output medium. Further, it has been found that shortening of the processing time can be achieved, because a sharpening processing is applied to viewing image referred image data having a comparatively small image size.  
     [0374] Embodiment 3  
     [0375] In the following, the embodiment 3 of this invention will be explained with reference to the drawings.  
     [0376] In addition, in this embodiment 3, it is shown an example using digital image data recorded by the image-capturing device  21  in the same way as explained in the above-mentioned embodiment 1.  
     Structure of Image Processing Apparatus  117   
     [0377]FIG. 25 is a block diagram showing the functional structure of an image processing apparatus  117  of this embodiment 3. As shown in FIG. 25, in this embodiment, the arrangement of the smoothing processor  113  and the sharpening processor  114  is different from that of the image processing apparatus  117  of the embodiment 1 shown in FIG. 16. That is, scene-referred image data d 4 , which have been generated by the application of a correction processing for image-capturing device characteristics to scene-referred raw data d 2 , are stored in the temporary storage  105 , are read out by the smoothing processor  113  of the optimization processor  116 , are subjected to a smoothing processing, are subjected to a sharpening processing by the sharpening processor  114 , and are subjected to an optimization processing in accordance with the output end by the viewing image referred image data generator  107 , to become viewing image referred image data d 5  generated.  
     [0378] In addition, because the other components of the structure of the image processing apparatus  117  of this embodiment 3 are the same as those of the above-mentioned embodiment 1, the same sign is attached to every structural component as that of the embodiment 1, and the explanation of the structure will be omitted.  
     Operation of Image Processing Apparatus  117   
     [0379]FIG. 26 is a flow chart showing an image data generation processing C to be practiced by the cooperation of the pertinent portions of the image processing apparatus  117 . In the following, the operation of the image processing apparatus  117  will be explained with reference to the drawings.  
     [0380] When a recording medium having recorded a data file of the structure shown in FIG. 9 is loaded, the digital image data file recorded in the recording medium is inputted by the input section  101  (step S 101 ). As regards the inputted digital image data, its content is analyzed by the header information analysis section  102  (step S 102 ), and is divided into scene-referred raw data d 2  (step S 103 ), correction data for image-capturing device characteristics d 1  (step S 104 ), and photographing information data d 3  (step S 105 ); the scene-referred raw data d 2  and the correction data for image-capturing device characteristics d 1  are outputted to the correction processor for image-capturing device characteristics  115 , and the photographing information data d 3  are outputted to the optimization processor  116 .  
     [0381] When the correction data for image-capturing device characteristics d 1  are inputted to the correction processor for image-capturing device characteristics  115 , the processing condition table  103   b  is referred to by the correction processor for apparatus characteristics  103   a , and processing conditions for generating scene-referred image data d 4  are determined. The scene-referred raw data d 2  are subjected to a correction processing for image-capturing device characteristics by the scene-referred image data generator  104  on the basis of the processing conditions determined by the correction processor for apparatus characteristics  103   a  (step S 106 ), to become scene-referred image data d 4  generated, which are stored in the temporary storage  105  (step S 107 ).  
     [0382] When the photographing information data d 3  are inputted to the optimization processor  116 , processing conditions for generating viewing image referred image data d 5  in accordance with the photographing conditions are determined by the photographing information data processor  106  on the basis of the photographing information data d 3 . The scene-referred image data d 4  are read out from the temporary storage  105 , are subjected to a smoothing processing by the smoothing processor  113  (step S 108 ), next are subjected to a sharpening processing by the sharpening processor  114  (step S 109 ), and further, on the basis of the processing conditions determined by the photographing information data processor  106  and the operation information inputted from the setting input section  109 , are subjected to an optimization processing by the viewing image referred image data generator  107  of the optimization processor  116  in accordance with the size of the device or the output medium of the output end (step S 110 ), to become viewing image referred image data d 5  generated, which are outputted to the device designated by the setting input section  109  (step S 111 ).  
     [0383] In addition, the block diagram shown in FIG. 25 and the flow chart shown in FIG. 26 is merely an example; as regards the processing by the smoothing processor  113 , it is appropriate so long as it is practiced in the process of generating viewing image referred image data d 5  from scene-referred image data d 4 , and as regards the processing by the sharpening processor  114 , it is appropriate so long as it is practiced after the processing by the smoothing processor  113 ; the process is not limited to the block diagram shown in FIG. 25 and the flow chart shown in FIG. 26. For example, it is also appropriate that scene-referred image data d 4 , after being subjected to several optimization processings in the viewing image referred image data generator  107 , are subjected to a smoothing processing in the smoothing processor  113 , and after that, are subjected to a sharpening processing in the sharpening processor  114 .  
     [0384] As explained in the foregoing, by the image processing apparatus  117 , a smoothing processing and a sharpening processing are applied to digital image data inputted from the image-capturing device  21  in the process of generating viewing image referred image data d 5 . For that reason, also in the case where scene-referred image data are inputted as input data, a suitable and rapid smoothing processing and sharpening processing can be practiced.  
     Structure of Image Recording Apparatus  201   
     [0385] Next, the structure and the operation of an image recording apparatus  201  of this embodiment 3 will be explained. In addition, because the outside structure and the internal structure of the image recording apparatus  201  and the structure of an image processor  270  of this embodiment 3 are the same as those of the above-mentioned embodiment 1 except that the order of data processing is different, the feature is to be explained in the explanation of the operation; further, every structural component is given the same sign as that of the embodiment 1, and it will be omitted to show the structure in the drawings and explain it.  
     [0386] Further, the image recording apparatus  201  of this embodiment 3 is one corresponding to an image recording apparatus as set forth in the structure (13) of this invention, the combination of the correction processor for image-capturing device characteristics  303   a  and the scene-referred image data generator  304  together has the function of a scene-referred image data generation means as set forth in the structure (13) of this invention, and the combination of the photographing information processor  306 , the viewing image referred image data processor  307 , the smoothing processor  313 , and the sharpening processor  314  together has the function of a viewing image referred image data generation means as set forth in the structure (13) of this invention. Further, the smoothing processor  313  has the function of a smoothing processor as set forth in the structure (13) of this invention, and the sharpening processor  314  has the function of a sharpening processor as set forth in the structure (13) of this invention.  
     Operation of Image Recording Apparatus  201   
     [0387]FIG. 27 is a flow chart showing an image data formation processing C to be practiced by the cooperation of the pertinent portions of the image processor  270 . In the following, the operation of every part of the image processor  270  will be explained with reference to the drawings.  
     [0388] Data are inputted from the image transfer means  230  or the communication means (input)  240  to the image processor  270  (step S 121 ), and when said inputted data are judged by the decoding processor for image data format  704  as a digital image data file obtained by the aforesaid image-capturing device  21  (step S 122 ), the content of the inputted digital image data file is analyzed by the header information analysis section  302  (step S 123 ), and is divided into scene-referred raw data d 2  (step S 124 ), correction data for image-capturing device characteristics d 1  (step S 125 ), and photographing information data d 3  (step S 126 ).  
     [0389] The correction data for image-capturing device characteristics d 1  are outputted to the correction processor for apparatus characteristics  303   a , and processing conditions for generating scene-referred image data are determined by the correction processor for apparatus characteristics  303   a  referring to the processing condition table  303   b . The scene-referred raw data d 2  are subjected to a correction processing for image-capturing device characteristics by the scene-referred image data generator  304 , on the basis of the processing conditions determined by the correction processor for apparatus characteristics  303   a  (step S 127 ), to become scene-referred image data d 4  generated, which are outputted to the smoothing processor  313  (step S 128 ).  
     [0390] When the scene-referred image data d 4  are subjected to a smoothing processing in the smoothing processor  313  (step S 129 ), the scene-referred image data are next subjected to a sharpening processing by the sharpening processor  314 , and are outputted to the viewing image referred image data generator  307  (step S 130 ).  
     [0391] When the photographing information data d 3  are inputted to the photographing information data processor  306 , processing conditions for generating viewing image referred image data d 5  in accordance with the photographing conditions are determined on the basis of the photographing information data d 3 . Further, on the basis of an instruction from the operation section  211  and the controller  207 , in the image adjustment processor  701 , image processing conditions for generating viewing image referred image data d 5  adapted to the output device and the output medium are determined. The scene-referred image data d 4  which have already been subjected to a smoothing processing and a sharpening processing are subjected to an optimization processing by the viewing image referred image data generator  307  in accordance with the size of the output device and the output medium of the output end, on the basis of the processing conditions determined by the photographing information data processor  306  and the image processing conditions determined by the image adjustment processor  701  (step S 131 ), to become viewing image referred image data generated, which are outputted to any one of the processors, the CRT-specific processor  706 , the printer-specific processor  707 , the printer-specific processor  708 , and the image data format generation processor  709  in accordance with the output end based on the operation at the operation section  211  (step S 132 ). The viewing image referred image data d 5  are subjected to a processing specific to the output end in accordance with it in the processor to which the data are outputted (step S 133 ), and are outputted to the output end designated by the operation section  211  (step S 134 ).  
     [0392] In addition, the flow chart shown in FIG. 27 is merely an example; as regards the processing by the smoothing processor  313 , it is appropriate so long as it is practiced in the process of generating viewing image referred image data d 5  from scene-referred image data d 4 ; further, as regards the processing by the sharpening processor  314 , it is appropriate so long as it is practiced after the processing by the smoothing processor  313 ; the process is not limited to the flow chart shown in FIG. 27. For example, it is also appropriate that scene-referred image data d 4 , after being subjected to several optimization processings in the viewing image referred image data generator  307 , are subjected to a smoothing processing in the smoothing processor  313 , and after that, are subjected to a sharpening processing in the sharpening processor  314 .  
     [0393] As explained in the foregoing, by the image recording apparatus  201 , a smoothing processing and a sharpening processing are applied to digital image data inputted from the image-capturing device  21  in the process of generating viewing image referred image data d 5 . For that reason, also in the case where scene-referred image data are inputted as input data, a suitable and rapid smoothing processing and sharpening processing can be practiced.  
     [0394] By the structure of this embodiment 3 explained in the above description, scene-referred raw data, scene-referred image data, and viewing image referred image data which have been subjected to a smoothing processing and a sharpening processing can be obtained. As regards the smoothing processing of this embodiment 3, for the purpose of reducing the processing time, it is applied to input data having comparatively large noises. It has been found out that a noise removal processing can be practiced in a most suitable way, because a smoothing processing is practiced after a correction processing based on correction data for apparatus characteristics is carried out. Further, it has been found that shortening of the processing time can be achieved.  
     [0395] Embodiment 4  
     [0396] In the following, the embodiment 4 of this invention will be explained with reference to the drawings.  
     [0397] In addition, in the embodiment 4, it is shown an example using digital image data recorded by the image-capturing device  21  in the same way as explained in the above-mentioned embodiment 1.  
     Structure of Image Processing Apparatus  
     [0398]FIG. 28 is a block diagram showing the functional structure of an image processing apparatus  117  of this embodiment 4. As shown in FIG. 28, in this embodiment, the arrangement of the smoothing processor  113  is different from that of the image processing apparatus  117  of the embodiment 1 shown in FIG. 16. That is, scene-referred image data d 4 , which have been generated by the application of a correction processing for image-capturing device characteristics to scene-referred raw data d 2 , are subjected to a smoothing processing by the smoothing processor  113 , to become smoothed scene-referred image data d 7  generated, which are stored in the temporary memory  105 .  
     [0399] In addition, because the other components of the structure of the image processing apparatus  117  of this embodiment 4 are the same as those of the above-mentioned embodiment 1, the same sign is attached to every structural component as that of the embodiment 1, and the explanation of the structure will be omitted.  
     Operation of Image Processing Apparatus  117   
     [0400]FIG. 29 is a flow chart showing an image data generation processing D to be practiced by the cooperation of the pertinent portions of the image processing apparatus  117 . In the following, the operation of the image processing apparatus  117  will be explained with reference to the drawings.  
     [0401] When a recording medium having recorded a data file of the structure shown in FIG. 9 is loaded, the digital image data file recorded in the recording medium is inputted by the input section  101  (step S 141 ). The content of inputted digital image data is analyzed by the header information analysis section  102  (step S 142 ), and is divided into scene-referred raw data d 2  (step S 143 ), correction data for image-capturing device characteristics d 1  (step S 144 ), and photographing information data d 3  (step S 145 ); the scene-referred raw data d 2  and the correction data for image-capturing device characteristics d 1  are outputted to the correction processor for image-capturing device characteristics  115 , and the photographing information data d 3  are outputted to the optimization processor  116 .  
     [0402] When the correction data for image-capturing device characteristics d 1  are inputted to the correction processor for image-capturing device characteristics  115 , the correction processor for apparatus characteristics  103   a  refers to the processing condition table  103   b , and processing conditions for generating scene-referred image data d 4  are determined. The scene-referred raw data d 2  are subjected to a correction processing for image-capturing device characteristics by the scene-referred image data generator  104  on the basis of the processing conditions determined by the correction processor for apparatus characteristics  103   a  (step S 146 ), to become scene-referred image data d 4  generated (step S 147 ). Then, scene-referred image data d 4  are subjected to a smoothing processing by the smoothing processor  113  (step S 148 ), to become smoothed scene-referred image data d 7  generated, which are stored in the temporary storage  105  (step S 149 ).  
     [0403] When the photographing information data d 3  are inputted to the optimization processor  116 , processing conditions for generating viewing image referred image data d 5  in accordance with the photographing conditions are determined by the photographing information data processor  106  on the basis of the photographing information data d 3 . The smoothed scene-referred image data d 7  are read out from the temporary storage  105 ; then, on the basis of the processing conditions determined by the photographing information data processor  106  and the operation information inputted from the setting input section  109 , the smoothed scene-referred image data d 7  are subjected to a sharpening processing in accordance with the size of the device or the output medium of the output end by the sharpening processor  114  (step S 150 ), and are subjected to an optimization processing by the viewing image referred image data generator  107  (step S 151 ), to become viewing image referred image data d 5  generated, which are outputted to the device designated by the setting input section  109  (step S 152 ).  
     [0404] In addition, the block diagram shown in FIG. 28 and the flow chart shown in FIG. 29 are merely an example; as regards the processing by the sharpening processor  114 , it is appropriate so long as it is practiced in the generation process of the viewing image referred image data d 5  from the smoothed scene-referred image data d 7 ; the process is not limited to the block diagram shown in FIG. 28 and the flow chart shown in FIG. 29. For example, it is also appropriate that scene-referred image data d 4 , after being subjected to several optimization processings in the viewing image referred image data processor  107 , are subjected to a sharpening processing in the sharpening processor  114 .  
     [0405] As explained in the foregoing, by the image processing apparatus  117 , because a smoothing processing is applied to scene-referred image data, even in the case where scene-referred image data d 4  are inputted as the input data, not to mention the case where scene-referred raw data d 2  are inputted, a suitable noise removal processing can be practiced.  
     [0406] Further, by the image processing apparatus  117 , a sharpening processing is applied to scene-referred image data d 4  in the generation process of viewing image referred image data d 5 . The generation process of viewing image referred image data d 5  is a process to apply image processing for optimizing digital image data in accordance with the output device and the output medium, and by the application of a sharpening processing in accordance with the output device and the output medium, it is possible to obtain a subjectively desirable image on the output medium.  
     Structure of Image Recording Apparatus  201   
     [0407] Next, the structure and the operation of an image recording apparatus  201  of this embodiment 4 will be explained. In addition, because the outside structure and the internal structure of the image recording apparatus  201  and the structure of an image processor  270  of this embodiment 4 are the same as those of the above-mentioned embodiment 1 except that the order of data processing is different, the feature is to be explained in the explanation of the operation; further, every structural component is given the same sign as that of the embodiment 1, and it will be omitted to show the structure in the drawings and explain it.  
     [0408] Further, the image recording apparatus  201  of this embodiment 4 is one corresponding to an image recording apparatus as set forth in the structure (14) of this invention, the combination of the correction processor for image-capturing device characteristics  303   a , the scene-referred image data generator  304 , and the smoothing processor  313  together has the function of a scene-referred image data generation means as set forth in the structure (14) of this invention, and the combination of the photographing information processor  306 , the viewing image referred image data processor  307 , and the sharpening processor  314  together has the function of a viewing image referred image data generation means as set forth in the structure (14) of this invention. Further, the smoothing processor  313  has the function of a smoothing processor as set forth in the structure (14) of this invention, and the sharpening processor  314  has the function of a sharpening processor as set forth in the structure (14) of this invention.  
     Operation of Image Recording Apparatus  201   
     [0409]FIG. 30 is a flow chart showing an image data formation processing E to be practiced by the cooperation of the pertinent portions of the image processor  270 . In the following, the operation of every part of the image processor  270  will be explained with reference to the drawings.  
     [0410] Data are inputted from the image transfer means  230  or the communication means (input)  240  to the image processor  270  (step S 161 ), and when said inputted data are judged by the decoding processor for image data format  704  as a digital image data file obtained by the aforesaid image-capturing device  21  (step S 162 ), the content of the inputted digital image data file is analyzed by the header information analysis section  302  (step S 163 ), and is divided into scene-referred raw data d 2  (step S 164 ), correction data for image-capturing device characteristics d 1  (step S 165 ), and photographing information data d 3  (step S 166 ).  
     [0411] The correction data for image-capturing device characteristics d 1  are outputted to the correction processor for apparatus characteristics  303   a , and processing conditions for generating scene-referred image data are determined by the correction processor for apparatus characteristics  303   a  referring to the processing condition table  303   b . The scene-referred raw data d 2  are subjected to a correction processing for image-capturing device characteristics by the scene-referred image data generator  304  on the basis of the processing conditions determined by the correction processor for apparatus characteristics  303   a  (step S 167 ), to become scene-referred image data d 4  generated (step S 168 ). After that, the scene-referred image data d 4  are subjected to a smoothing processing in the smoothing processor  313  (step S 169 ), to become smoothed scene-referred image data generated, which are outputted to the viewing image referred image data generator  307  (step S 170 ).  
     [0412] When the photographing information data d 3  are inputted to the photographing information data processor  306 , processing conditions for generating viewing image referred image data d 5  in accordance with the photographing conditions are determined on the basis of the photographing information data d 3 . Further, on the basis of an instruction from the operation section  211  and the controller  207 , in the image adjustment processor  701 , image processing conditions for generating viewing image referred image data d 5  adapted to the output device and the output medium are determined. The smoothed scene-referred image data d 7  which have already been subjected to a smoothing processing, on the basis of the processing conditions determined by the photographing information data processor  306  and the image processing conditions determined by the image adjustment processor  701 , are subjected to a sharpening processing by the sharpening processor  314  in accordance with the size of the device and the output medium of the output end (step S 171 ), and are subjected to an optimization processing by the viewing image referred image data generator  307  (step S 172 ), to become viewing image referred image data generated (step S 173 ).  
     [0413] The generated viewing image referred image data d 5  are outputted to any one of the processors, the CRT-specific processor  706 , the printer-specific processor  707 , the printer-specific processor  708 , and the image data format generation processor  709  in accordance with the output end based on the operation at the operation section  211 , are subjected to a processing specific to the output end in accordance with it in the processor to which the data are outputted (step S 174 ), and are outputted to the output end designated by the operation section  211  (step S 175 ).  
     [0414] In addition, the flow chart shown in FIG. 27 is merely an example; as regards the processing by the sharpening processor  314 , it is appropriate so long as it is practiced in the process of generating viewing image referred image data d 5  from smoothed scene-referred image data d 7 ; the process is not limited to the flow chart shown in FIG. 30. For example, it is also appropriate that smoothed scene-referred image data d 7 , after being subjected to several optimization processings in the viewing image referred image data generator  307 , are subjected to a sharpening processing in the sharpening processor  314 .  
     [0415] As explained in the foregoing, by the image recording apparatus  201 , because a smoothing processing is applied to scene-referred image data d 4 , even in the case where scene-referred image data d 4  are inputted as the input data, not to mention the case where scene-referred raw data d 2  are inputted, a suitable noise removal processing can be practiced.  
     [0416] Further, by the image recording apparatus  201 , a sharpening processing is applied to smoothed scene-referred image data d 7  in the generation process of viewing image referred image data d 5 . The generation process of viewing image referred image data is a process to apply image processing for optimizing digital image data in accordance with the output device and the output medium, and by the application of a sharpening processing in accordance with the output device and the output medium, it is possible to obtain a subjectively desirable image on the output medium.  
     [0417] By the structure of this embodiment 4, it is possible to obtain scene-referred raw data, scene-referred image data which have already been subjected to a smoothing processing, and viewing image referred image data which have already been subjected to a smoothing processing and a sharpening processing. As regards a smoothing processing of this embodiment 4, for the purpose of shortening the processing time, it is applied to input data having comparatively large noises. It has been found that, in this embodiment 4, because a smoothing processing is practiced after a correction process for image-capturing device characteristics is carried out, a noise removal processing can be practiced in a most suitable way. In this embodiment 4, because a sharpening processing is practiced in the generation process of viewing image referred image data, a sharpening processing is practiced before the practice of data compression. It has been found that, for this reason, a sharpening processing which is excellent in the sense of total balance with the color gamut and gradation of an image can be practiced and a good image can be obtained on output media of various kinds.  
     [0418] Embodiment 5  
     [0419] In the following, the embodiment 5 of this invention will be explained with reference to the drawings.  
     [0420] In addition, in this embodiment 5, it is shown an example using digital image data recorded by the image-capturing device  21  in the same way as explained in the above-mentioned embodiment 1.  
     Structure of Image Processing Apparatus  117   
     [0421]FIG. 31 is a block diagram showing the functional structure of an image processing apparatus  117  of this embodiment 5. As shown in FIG. 31, in this embodiment 5, the arrangement of the smoothing processor  113  and the sharpening processor  114  is different from that of the image processing apparatus  117  of the embodiment 1 shown in FIG. 16. That is, scene-referred image data d 4 , which have been generated by the application of a correction processing for image-capturing device to scene-referred raw data d 2 , are subjected to a smoothing processing by the smoothing processor  113 , to become smoothed scene-referred image data d 7  generated, which are stored in the temporary storage  105 . The smoothed scene-referred image data d 7  are read out by the viewing image referred image data generator  107 , and are subjected to an optimization processing in accordance with the output end, to become viewing image referred image data d 5  generated. After that, a sharpening processing is applied to the viewing image referred image data d 5  by the sharpening processor  114 , to generate sharpened viewing image referred image data d 6 .  
     [0422] In addition, because the other components of the structure of the image processing apparatus  117  of this embodiment 5 are the same as those of the above-mentioned embodiment 1, the same sign is attached to every structural component as that of the embodiment 1, and the explanation of the structure will be omitted.  
     Operation of Image Processing Apparatus  117   
     [0423]FIG. 32 is a flow chart showing an image data generation processing E to be practiced by the cooperation of the pertinent portions of the image processing apparatus  117 . In the following, the operation of the image processing apparatus  117  will be explained with reference to the drawings.  
     [0424] When a recording medium having recorded a data file of the structure shown in FIG. 9 is loaded, the digital image data file recorded in the recording medium is inputted by the input section  101  (step S 181 ). The content of inputted digital image data is analyzed by the header information analysis section  102  (step S 182 ), and is divided into scene-referred raw data d 2  (step S 183 ), correction data for image-capturing device characteristics d 1  (step S 184 ), and photographing information data d 3  (step S 185 ); the scene-referred raw data d 2  and the correction data for image 164  capturing device characteristics d 1  are outputted to the correction processor for image-capturing device characteristics  115 , and the photographing information data d 3  are outputted to the optimization processor  116 .  
     [0425] When the correction data for image-capturing device characteristics d 1  are inputted to the correction processor for image-capturing device characteristics  115 , the correction processor for apparatus characteristics  103   a  refers to the processing condition table  103   b , and processing conditions for generating scene-referred image data d 4  are determined. The scene-referred raw data d 2  are subjected to a correction processing for image-capturing device characteristics by the scene-referred image data generator  104  on the basis of the processing conditions determined by the correction processor for apparatus characteristics  103   a  (step S 186 ), to become scene-referred image data d 4  generated (step S 187 ). Then, the generated scene-referred image data d 4  are subjected to a smoothing processing by the smoothing processor  113  (step S 188 ), to become smoothed scene-referred image data d 7  generated, which are stored in the temporary storage  105  (step S 189 ).  
     [0426] When the photographing information data d 3  are inputted to the optimization processor  116 , processing conditions for generating viewing image referred image data d 5  in accordance with the photographing conditions are determined by the photographing information data processor  106  on the basis of the photographing information data d 3 . The smoothed scene-referred image data d 7  are read out to the viewing image referred image data generator  107  of the optimization processor  116 ; then, on the basis of the processing conditions determined by the photographing information data processor  106  and the operation information inputted from the setting input section  109 , the smoothed scene-referred image data d 7  are subjected to an optimization processing by the viewing image referred image data generator  107  in accordance with the size of the device and the output medium of the output end (step S 190 ), to become viewing image referred image data d 5  generated (step S 191 ) The generated viewing image referred image data d 5  are subjected to a sharpening processing by the sharpening processor  114  (step S 192 ), to become sharpened viewing image referred image data d 6  generated, which are outputted to the device designated by the setting input section  109  (step S 193 ).  
     [0427] As explained in the foregoing, by the image processing apparatus  117 , because scene-referred image data d 4  are subjected to a smoothing processing, even in the case where scene-referred image data d 4  are inputted as the input data, not to mention the case where scene-referred raw data d 2  are inputted, a suitable noise removal processing can be carried out.  
     [0428] Further, by the image processing apparatus  117 , a sharpening processing is practiced after the generation of viewing image referred image data d 5 . For that reason, in a case such that viewing image referred image data d 5  which have been generated through the application of an optimization processing based on the kind of a specified output device and output medium are to be outputted to output devices and output media of the same kind with the size of the output media changed, the optimization processing becomes unnecessary and the processing load can be lightened.  
     Structure of Image Recording Apparatus  201   
     [0429] Next, the structure and the operation of an image recording apparatus  201  of this embodiment 5 will be explained. In addition, because the outside structure and the internal structure of the image recording apparatus  201  and the structure of an image processor  270  of this embodiment 5 are the same as those of the above-mentioned embodiment 1 except that the order of data processing is different, the feature is to be explained in the explanation of the operation; further, every structural component is given the same sign as that of the embodiment 1, and it will be omitted to show the structure in the drawings and explain it.  
     [0430] Further, the image recording apparatus  201  of this embodiment 5 is one corresponding to an image recording apparatus as set forth in the structure (15) of this invention, the combination of the correction processor for image-capturing device characteristics  303   a , the scene-referred image data generator  304 , and the smoothing processor  313  together has the function of a scene-referred image data generation means as set forth in the structure (15) of this invention, and the combination of the photographing information processor  306 , the viewing image referred image data processor  307 , and the sharpening processor  314  together has the function of a viewing image referred image data generation means as set forth in the structure (15) of this invention. Further, the smoothing processor  313  has the function of a smoothing processor as set forth in the structure (15) of this invention, and the sharpening processor  314  has the function of a sharpening processor as set forth in the structure (15) of this invention.  
     Operation of Image Recording Apparatus  201   
     [0431]FIG. 33 is a flow chart showing an image data formation processing E to be practiced by the cooperation of the pertinent portions of the image processor  270 . In the following, the operation of every part of the image processor  270  will be explained with reference to the drawings.  
     [0432] Data are inputted from the image transfer means  230  or the communication means (input)  240  to the image processor  270  (step S 201 ), and when said inputted data are judged by the decoding processor for image data format  704  as a digital image data file obtained by the aforesaid image-capturing device  21  (step S 202 ), the content of the inputted digital image data file is analyzed by the header information analysis section  302  (step S 203 ), and is divided into scene-referred raw data d 2  (step S 204 ), correction data for image-capturing device characteristics d 1  (step S 205 ), and photographing information data d 3  (step S 206 ).  
     [0433] The correction data for image-capturing device characteristics d 1  are outputted to the correction processor for apparatus characteristics  303   a , and processing conditions for generating scene-referred image data are determined by the correction processor for apparatus characteristics  303   a  referring to the processing condition table  303   b . The scene-referred referred raw data d 2  are subjected to a correction processing for image-capturing device characteristics by the scene-referred image data generator  304  on the basis of the processing conditions determined by the correction processor for apparatus characteristics  303   a  (step S 207 ), to become scene-referred image data d 4  generated (step S 208 ). After that, the generated scene-referred image data d 4  are subjected to a smoothing processing in the smoothing processor  313  (step S 209 ), to become smoothed scene-referred image data d 7  generated, which are outputted to the viewing image referred image data generator  307  (step S 210 ).  
     [0434] When the photographing information data d 3  are inputted to the photographing information data processor  306 , processing conditions for generating viewing image referred image data d 5  in accordance with the photographing conditions are determined on the basis of the photographing information data d 3 . Further, on the basis of an instruction from the operation section  211  and the controller  207 , in the image adjustment processor  701 , image processing conditions for generating viewing image referred image data d 5  adapted to the output device and the output medium are determined. On the basis of the processing conditions determined by the photographing information data processor  306  and the image processing conditions determined by the image adjustment processor  701 , the smoothed scene-referred image data d 7  which have already been subjected to a smoothing processing, are subjected to an optimization processing by the viewing image referred image data generator  307  in accordance with the size of the device and the output medium of the output end, (step S 211 ), to become viewing image referred image data d 5  generated (step S 212 ).  
     [0435] The generated viewing image referred image data d 5  are subjected to a sharpening processing by the sharpening processor  314  (step S 213 ), to become sharpened viewing image referred image data d 6 , and are outputted to any one of the processors, the CRT-specific processor  706 , the printer-specific processor  707 , the printer-specific processor  708 , and the image data format generation processor  709  in accordance with the output end based on the operation at the operation section  211  (step S 214 ). The sharpened viewing image referred image data d 6  are subjected to a processing specific to the output end in accordance with it in the processor to which the data are outputted (step S 215 ), and are outputted to the output end designated by the operation section  211  (step S 216 ).  
     [0436] As explained in the foregoing, by the image recording apparatus  201 , because a smoothing processing is applied to scene-referred image data, even in the case where scene-referred image data d 4  are inputted as the input data, not to mention the case where scene-referred raw data d 2  are inputted, a suitable noise removal processing can be practiced.  
     [0437] Further, by the image recording apparatus  201 , a sharpening processing is practiced after the generation of viewing image referred image data d 5 . For that reason, in a case such that viewing image referred image data d 5  which have been generated through the application of an optimization processing based on the kind of a specified output device and output medium are to be outputted to output devices and output media of the same kind with the size of the output media changed, the optimization processing becomes unnecessary, which makes it possible to lighten the processing load.  
     [0438] In this way, by the structure of this embodiment 5, it is possible to obtain scene-referred raw data, scene-referred image data which have already been subjected to a smoothing processing, and viewing image referred image data which have already been subjected to a smoothing processing and a sharpening processing. As regards a smoothing processing of this embodiment 5, for the purpose of shortening the processing time, it is applied to input data having comparatively large noises. It has been found that, in this embodiment 5, because a smoothing processing is practiced after a correction processing using correction data for image-capturing device characteristics and a correction processing using photographing information data are carried out, a noise removal processing can be practiced in a most suitable way. In this embodiment 5, because a sharpening processing is practiced in a process later than the generation of viewing image referred image data, a sharpening processing is practiced on the basis of the output information. It has been found that, for this reason, an image having a very excellent sharpness can be obtained on a specified output medium. Further, it has been found that, because a sharpening processing is applied to viewing image referred image data of a comparatively small image size, shortening of the processing time can be achieved.  
     [0439] Further, in the embodiment 1 to embodiment 5, a smoothing processing is practiced by the use of a low-pass filter with the mask size, mask shape, and the threshold value changed on the basis of the characteristic of the noise; therefore, it is possible to prevent the lowering of image quality, and also it is possible to make the speed of processing higher. Further, because the amount of edge enhancement in a sharpening processing is adjusted in accordance with the kind and size of the output medium, size of the main photographic object in an image, and the photographed scene, a subjectively more desirable image quality can be obtained.  
     [0440] In addition, in the above-mentioned embodiment 1 to embodiment 5, explanation has been given for the mode of practice where digital image data having a data structure shown in FIG. 9 recorded by the image-capturing device  21  are inputted; however, it is possible to obtain digital image data having the structure shown in FIG. 8, and FIG. 10 to FIG. 13 by it that scene-referred image data and viewing image referred image data are generated on the basis of the captured-image data in the image processor  7  of the image-capturing device  21 , and to image data having been generated by the use of the difference between the viewing image referred image data and the scene-referred image data generated, correction data for image-capturing device characteristics and photographing information data are attached.  
     [0441] In the case where a file having a data structure as shown in FIG. 10 or FIG. 11 in the image processing apparatus  117  or in the image recording apparatus  201 , the header information analysis section  102  ( 302 ) divides the inputted digital image data into correction data for image-capturing device characteristics d 1 , scene-referred image data d 4 , and photographing information data d 3  (in the case of FIG. 11); generation process of scene-referred image data is omitted, and viewing image referred image data d 5  are generated from scene-referred image data d 4  in the viewing image referred image data generation process.  
     [0442] In the case where a file having a data structure as shown in FIG. 12 or FIG. 13 is inputted in the image processing apparatus  117  or in the image recording apparatus  201 , the header information analysis section  102  ( 302 ) divides the inputted digital image data into correction data for image-capturing device characteristics d 1 , photographing information data d 3  (in the case of FIG. 13), viewing image referred image data d 5 , and difference data d 8 , and in the scene-referred image data generation process, on the basis of the correction data for image-capturing device characteristics d 1 , the difference data d 8 , and the photographing information data d 3  (in the case of FIG. 13), a correction processing for the transformation characteristics specific to the image-capturing device is applied to the viewing image referred image data d 5 , to generate scene-referred image data d 4 . Further, viewing image referred image data d 5  are generated from the scene-referred image data d 4  in the viewing image referred image data generation process. Besides, in the case where the image processing apparatus  117  or the image recording apparatus  201  is an apparatus which generates smoothed scene-referred image data d 7  after generating scene-referred image data d 4 , it generates viewing image referred image data d 5  from the smoothed scene-referred image data d 7  in the viewing image referred image data generation process.  
     [0443] As described in the foregoing, the present invention can use data outputted from image-capturing device of three types, namely, a type outputting scene-referred raw data, a type outputting scene-referred image data, and a type outputting viewing image referred image data.  
     [0444] In addition to the above description, the detailed structure and the detailed operation of the apparatus making up the present invention can be altered suitably within a range not departing from the spirit of this invention.  
     EFFECT OF THIS INVENTION  
     [0445] As described in the foregoing, it has become possible to obtain an image forming method, an image processing apparatus, and an image recording apparatus to make it possible to rapidly obtain image quality having very small amount of noises and an excellent sharpness, wherein scene-referred raw data which are unprocessed direct output signals obtained by the recording of information true to the photographic object with neither image processing such as gradation transformation, sharpness enhancement, chroma enhancement to alter the data content for the purpose of improving the effect at the time of image appreciation, nor processing to map signal strength data of the color channels based on the spectral sensitivity specific to the image sensor to a standardized color space such as the above-mentioned rimm rgb, or srgb applied, scene-referred image data which are obtained by the application of correction processing to unprocessed output signals of all the image sensors to become the object by the use of correction parameters for each photoelectric conversion function of the image sensor set beforehand for said scene-referred raw data, and viewing image referred image data which have been subjected to processings to make it possible to obtain an image on a display device, or an output medium.