Patent Publication Number: US-2020304790-A1

Title: Image processing apparatus and method

Description:
CROSS REFERENCE TO PRIOR APPLICATION 
     This application is a continuation of U.S. patent application Ser. No. 16/325,312 (filed on Feb. 13, 2019), which is a National Stage Patent Application of PCT International Patent Application No. PCT/JP2017/033527 (filed on Sep. 15, 2017) under 35 U.S.C. § 371, which claims priority to Japanese Patent Application No. 2016-193687 (filed on Sep. 30, 2016), which are all hereby incorporated by reference in their entirety. 
    
    
     TECHNICAL FIELD 
     The present disclosure relates to an image processing apparatus and method, and particularly to an image processing apparatus and method that make it possible to suppress reduction of the encoding efficiency. 
     BACKGROUND ART 
     In the past, adaptive primary transforms (AMT: Adaptive Multiple Core Transforms) have been disclosed in which, for each of a primary transform PTor in a horizontal direction (also called primary horizontal transform) and a primary transform PTver in a vertical direction (also called primary vertical transform), a primary transform is adaptively selected from a plurality of different orthogonal transforms (for example, refer to NPL 1). 
     It is to be noted that also it is disclosed in NPL 1 that, for each of a horizontal direction (x direction) and a vertical direction (y direction), a transform set TransformSet including orthogonal transforms that become candidates for a primary transform is (uniquely) determined (selected) on the basis of a correspondence table (intra prediction mode information) between mode information and transform sets. Also it is disclosed that a definition of a transform set is determined on the basis of a transform block size and mode information (for example, refer to NPL 2). 
     CITATION LIST 
     Non Patent Literature 
     
         
         [NPL 1] 
         JVET-C1001, Algorithm description of Joint Exploration Test Model 3, published 2016 Jul. 2, url:http://phenix.int-evry.fr/jvet/doc_end_user/documents/3_Geneva/wg11/JVET-C1001-v3.zip 
         [NPL 2] 
         JVET-C0022, Proposed improvements to the Adaptive multiple core transform, published 2016 May 16, url:http://phenix.int-evry.fr/jvet/doc_end_user/documents/3_Geneva/wg11/JVET-C0022-v4.zip 
       
    
     SUMMARY 
     Technical Problem 
     However, the existing method has a limitation that one-dimensional transform skip can be selected only in the case of a specific transform block size and intra prediction mode number. Accordingly, in the case where it is better in the point of view of rate distortion to select one-dimensional transform skip of skipping an orthogonal transform in a horizontal or vertical direction than to perform two-dimensional orthogonal transform, since the encoder side cannot select the one-dimensional transform skip, there is the possibility that the encoding efficiency may reduce. 
     The present disclosure has been made in view of such a situation as described above and makes it possible to suppress the reduction of the encoding efficiency. 
     Solution to Problem 
     An image processing apparatus of a first aspect of the present technology is an image processing apparatus including a decoding section configured to decode encoded data, an inverse primary vertical transform controlling section configured to control, based on a value of a transform skip identifier obtained by the decoding of the encoded data by the decoding section, execution of an inverse primary vertical transform that is an inverse primary transform in a vertical direction for transform coefficient data transformed from image data, and an inverse primary horizontal transform controlling section configured to control, based on the value of the transform skip identifier, execution of an inverse primary horizontal transform that is an inverse primary transform in a horizontal direction for the coefficient data transformed from the image data. 
     The inverse primary vertical transform controlling section can control the execution of the inverse primary vertical transform such that, where the transform skip identifier indicates that a one-dimensional transform in the vertical direction is not to be skipped, the inverse primary vertical transform for the transform coefficient data is executed, but where the transform skip identifier indicates that a one-dimensional transform in the vertical direction is to be skipped, the inverse primary vertical transform for the transform coefficient data is omitted. 
     The inverse primary horizontal transform controlling section can control the execution of the inverse primary horizontal transform such that, where the transform skip identifier indicates that a one-dimensional transform in the horizontal direction is not to be skipped, the inverse primary horizontal transform for the transform coefficient data is executed, but where the transform skip identifier indicates that a one-dimensional transform in the horizontal direction is to be skipped, the inverse primary horizontal transform for the transform coefficient data is omitted. 
     The image processing apparatus can further include a selection section configured to select an orthogonal transform that is to be applied to the inverse primary vertical transform and the inverse primary horizontal transform. 
     The selection section can select an orthogonal transform to be applied as the inverse primary vertical transform based on a vertical transform set identifier and a primary vertical transform designation flag obtained by the decoding of the encoded data by the decoding section; and select an orthogonal transform to be applied as the inverse primary horizontal transform based on a horizontal transform set identifier and a primary horizontal transform designation flag obtained by the decoding of the encoded data by the decoding section. 
     The decoding section can derive the primary vertical transform designation flag and the primary horizontal transform designation flag from a primary transform identifier in response to the value of the transform skip identifier. 
     The decoding section can derive, where the transform skip identifier indicates that a two-dimensional transform is not to be skipped, the primary vertical transform designation flag and the primary horizontal transform designation flag by processing the primary transform identifier as a 2-bit bin string, and can derive, where the transform skip identifier indicates that a one-dimensional transform in the vertical direction or the horizontal direction is not to be skipped, the primary vertical transform designation flag and the primary horizontal transform designation flag by processing the primary transform identifier as a 1-bit bin string. 
     The decoding section can decode the primary vertical transform designation flag and the primary horizontal transform designation flag included in the encoded data. 
     Where the transform skip identifier indicates that a one-dimensional transform in the vertical direction or the horizontal direction or a two-dimensional transform is not to be skipped, the decoding section can omit decoding of a secondary transform identifier and set the secondary transform identifier to a value that indicates that a secondary transform is not to be performed. 
     An image processing method of the first aspect of the present technology is an image processing method including decoding encoded data, controlling, based on a value of a transform skip identifier obtained by the decoding of the encoded data, execution of an inverse primary vertical transform that is an inverse primary transform in a vertical direction for transform coefficient data transformed from image data, and controlling, based on the value of the transform skip identifier, execution of an inverse primary horizontal transform that is an inverse primary transform in a horizontal direction for the coefficient data transformed from the image data. 
     An image processing apparatus of a second aspect of the present technology is an image processing apparatus including a primary horizontal transform controlling section configured to control execution of a primary horizontal transform that is a primary transform in a horizontal direction for residual data between an image and a prediction image based on a value of a transform skip identifier, a primary vertical transform controlling section configured to control, based on a value of the transform skip identifier, execution of a primary vertical transform that is a primary transform in a vertical direction for the residual data between the image and the prediction image, and an encoding section configured to encode the transform skip identifier. 
     The primary horizontal transform controlling section can control the execution of the primary horizontal transform such that, where the transform skip identifier indicates that a one-dimensional transform in the horizontal direction is not to be skipped, the primary horizontal transform for the residual data is executed, but where the transform skip identifier indicates that a one-dimensional transform in the horizontal direction is to be skipped, the primary horizontal transform for the residual data is omitted. 
     The primary vertical transform controlling section can control the execution of the primary horizontal transform such that, where the transform skip identifier indicates that a one-dimensional transform in the vertical direction is not to be skipped, the primary vertical transform for the residual data is executed, but where the transform skip identifier indicates that a one-dimensional transform in the vertical direction is to be skipped, the primary vertical transform for the residual data is omitted. 
     The image processing apparatus can further include a selection section configured to select an orthogonal transform that is to be applied to the primary horizontal transform and the inverse primary vertical transform. 
     The selection section can select an orthogonal transform to be applied as the primary horizontal transform based on a horizontal transform set identifier and a primary horizontal transform designation flag, and can select an orthogonal transform to be applied as the primary vertical transform based on a vertical transform set identifier and a primary vertical transform designation flag. 
     The encoding section can derive a primary transform identifier from the primary horizontal transform designation flag and the primary vertical transform designation flag in response to the value of the transform skip identifier. 
     The encoding section can derive, where the transform skip identifier indicates that two-dimensional transform is not to be skipped, the primary transform identifier of a 2-bit bin string using the primary horizontal transform designation flag and the primary vertical transform designation flag, and can derive, where the transform skip identifier indicates that a one-dimensional transform in the vertical direction or the horizontal direction is not to be skipped, the primary transform identifier of a 1-bit bin string using the primary horizontal transform designation flag or the primary vertical transform designation flag. 
     The encoding section can encode the primary horizontal transform designation flag and the primary vertical transform designation flag. 
     Where the transform skip identifier indicates that a one-dimensional transform in the vertical direction or the horizontal direction or a two-dimensional transform is not to be skipped, the encoding section can omit encoding of a secondary transform identifier. 
     An image processing method of the second aspect of the present technology is an image processing method including controlling execution of a primary horizontal transform that is a primary transform in a horizontal direction for residual data between an image and a prediction image based on a value of a transform skip identifier, controlling, based on a value of the transform skip identifier, execution of a primary vertical transform that is a primary transform in a vertical direction for the residual data between the image and the prediction image, and encoding the transform skip identifier. 
     In the image processing apparatus and method of the first aspect of the present technology, encoded data is decoded, and based on a value of a transform skip identifier obtained by the decoding of the encoded data, execution of an inverse primary vertical transform that is an inverse primary transform in the vertical direction for transform coefficient data transformed from image data is controlled. Further, based on the value of the transform skip identifier, execution of an inverse primary horizontal transform that is an inverse primary transform in the horizontal direction for the coefficient data transformed from the image data is controlled. 
     In the image processing apparatus and method of the second aspect of the present technology, execution of a primary horizontal transform that is a primary transform in a horizontal direction for residual data between an image and a prediction image is controlled based on a value of a transform skip identifier, and based on a value of the transform skip identifier, execution of a primary vertical transform that is a primary transform in a vertical direction for the residual data between the image and the prediction image is executed. Then, the transform skip identifier is encoded. 
     Advantageous Effect of Invention 
     According to the present disclosure, an image can be processed. Especially, reduction of the encoding efficiency can be suppressed. 
    
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         FIG. 1  is a view depicting a correspondence relationship between transform sets and orthogonal transforms to be selected. 
         FIG. 2  is a view depicting a correspondence relationship between types of orthogonal transform and functions to be used. 
         FIG. 3  is a view depicting a correspondence relationship between transform sets and prediction modes. 
         FIG. 4  is a view depicting a correspondence relationship between transform sets and block sizes. 
         FIG. 5  is a view depicting semantics of transform skip identifiers. 
         FIG. 6  is an explanatory view illustrating an overview of recursive block segmentation of a CU. 
         FIG. 7  is an explanatory view illustrating setting of a PU to the CU depicted in  FIG. 6 . 
         FIG. 8  is an explanatory view illustrating setting of a TU to the CU depicted in  FIG. 6 . 
         FIG. 9  is an explanatory view illustrating a scanning order of CUs/PUs. 
         FIG. 10  is a block diagram depicting a principal configuration example of an image decoding apparatus. 
         FIG. 11  is a block diagram depicting a principal configuration example of an inverse transform section. 
         FIG. 12  is a flow chart illustrating an example of a flow of an image decoding process. 
         FIG. 13  is a flow chart illustrating an example of a flow of an inverse transform process. 
         FIG. 14  is a flow chart illustrating an example of a flow of an inverse primary transform selection process. 
         FIG. 15  is a block diagram depicting a principal configuration example of a decoding section. 
         FIG. 16  is a flow chart illustrating an example of a flow of a decoding process. 
         FIG. 17  is a view depicting an example of syntax. 
         FIG. 18  is a block diagram depicting a principal configuration example of a decoding section. 
         FIG. 19  is a flow chart illustrating an example of a flow of a decoding process. 
         FIG. 20  is a flow chart illustrating an example of a flow of a primary vertical/horizontal transform designation flag derivation process. 
         FIG. 21  is a block diagram depicting a principal configuration example of a decoding section. 
         FIG. 22  is a flow chart illustrating an example of a flow of a decoding process. 
         FIG. 23  is a view depicting an example of syntax. 
         FIG. 24  is a block diagram depicting a principal configuration example of a decoding section. 
         FIG. 25  is a view depicting an example of syntax. 
         FIG. 26  is a block diagram depicting a principal configuration example of an inverse transform section. 
         FIG. 27  is a block diagram depicting a principal configuration example of an image encoding apparatus. 
         FIG. 28  is a block diagram depicting a principal configuration example of a transform section. 
         FIG. 29  is a flow chart illustrating an example of a flow of an image encoding process. 
         FIG. 30  is a flow chart illustrating an example of a flow of a transform process. 
         FIG. 31  is a view illustrating a flow of a primary transform selection process. 
         FIG. 32  is a block diagram depicting a principal configuration example of an encoding section. 
         FIG. 33  is a flow chart illustrating an example of a flow of an encoding process. 
         FIG. 34  is a block diagram depicting a principal configuration example of an encoding section. 
         FIG. 35  is a flow chart illustrating an example of a flow of an encoding process. 
         FIG. 36  is a flow chart illustrating an example of a flow of a primary transform identifier derivation process. 
         FIG. 37  is a block diagram depicting a principal configuration example of an encoding section. 
         FIG. 38  is a flow chart illustrating an example of a flow of an encoding process. 
         FIG. 39  is a block diagram depicting a principal configuration example of an encoding section. 
         FIG. 40  is a block diagram depicting a principal configuration example of a transform section. 
         FIG. 41  is a block diagram depicting a principal configuration example of a computer. 
         FIG. 42  is a block diagram depicting an example of general configuration of a television apparatus. 
         FIG. 43  is a block diagram depicting an example of general configuration of a portable telephone set. 
         FIG. 44  is a block diagram depicting an example of general configuration of a recording and reproduction apparatus. 
         FIG. 45  is a block diagram depicting an example of general configuration of an imaging apparatus. 
         FIG. 46  is a block diagram depicting an example of general configuration of a video set. 
         FIG. 47  is a block diagram depicting an example of general configuration of a video processor. 
         FIG. 48  is a block diagram depicting another example of general configuration of a video processor. 
         FIG. 49  is a block diagram depicting an example of general configuration of a network system. 
     
    
    
     DESCRIPTION OF EMBODIMENTS 
     In the following, modes for carrying out the present disclosure (hereinafter referred to as embodiments) are described. It is to be noted that the description is given in the following order. 
     1. First Embodiment (image decoding apparatus) 
     2. Second Embodiment (decoding of identifier) 
     3. Third Embodiment (encoding of primary transform designation flag) 
     4. Fourth Embodiment (scaling) 
     5. Fifth Embodiment (image encoding apparatus) 
     6. Sixth Embodiment (decoding of identifier) 
     7. Seventh Embodiment (encoding of primary transform designation flag) 
     8. Eighth Embodiment (scaling) 
     9. Ninth Embodiment (others) 
     First Embodiment 
     &lt;Primary Transform&gt; 
     In the test model (JEM3 (Joint Exploration Test Model 3)) disclosed in NPL 1, for the object of enhancement of the encoding efficiency of a high resolution image of 4K or the like, the maximum size of the CTU size is expanded from 128×128 to 256×256. Further, as a structure for block segmentation, a binary tree in horizontal/vertical directions is introduced in addition to existing quad-tree segmentation, and together with this, also a non-rectangular transform block is introduced in addition to a rectangular transform block. 
     Further, in JEM3, adaptive primary transforms (AMT (Adaptive Multiple Core Transforms)) are disclosed in which, for each TU unit, a primary transform is adaptively selected from among a plurality of different orthogonal transforms for each of a primary transform PThor in a horizontal direction (also referred to as primary horizontal transform) and a primary transform PTver in a vertical direction (also referred to as primary vertical transform). 
     More particularly, in the case where an adaptive primary transform flag apt_flag (also called amt_flag, cu_pt_flag or emt_flag) indicative of whether or not an adaptive primary transform is to be carried out, for example, in a TU unit is 0 (false), as a primary transform, DCT-II or DST-VII is determined (uniquely) by mode information. 
     In contrast, for example, in the case where the adaptive primary transform flag apt_flag is 1 (true), a transform set TransformSet including orthogonal transforms that become candidates for primary transforms in regard to a horizontal direction (x direction) and a vertical direction (y direction) is selected from among three candidates (Transform SetIdx=0 to 2) as depicted in  FIG. 1 . DSI (Discrete Sine Transform)-VII, DCT (Discrete Cosine Transform)-VIII and so forth depicted in  FIG. 1  indicate types of orthogonal transforms, and such functions as in a table depicted in  FIG. 2  are used for them. 
     Selection (determination) of a transform set TransformSet is performed on the basis of an intra prediction mode (Intra Mode) as in a table depicted in  FIG. 3 . For example, the selection (determination) of a transform set TransformSet is carried out such that a transform set identifier TransformSetIdx that designates a transform set TransformSet corresponding to each transform set TransformSet{H, V} is set to the transform set TransformSet{H, V} as indicated by the following expression (1) and (2). 
       TransformSet H =LUT_IntraModeToTransformSet [Intramode][ H (=0)]   (1)
 
       TransformSet V =LUT_IntraModeToTransformSet [Intramode][ V (=1)]   (2)
 
     Here, TransformSetH indicates a transform set of a primary horizontal transform PThor while TransformSetV indicates a transform set of a primary vertical transform PTver, and a lookup table LUT_IntraModeToTransformSet indicates the correspondence table of  FIG. 3 . The first array of the lookup table LUT_IntraModeToTransformSet[ ][ ] takes the intra prediction mode IntraMode as an argument, and the second array takes {H=0, V=1} as an argument. 
     For example, in the case of the intra prediction mode number 9 (IntraMode==9), as the transform set TransformSetH of a primary horizontal transform PThor (also referred to as primary horizontal transform set), the transform set of the transform set identifier TransformSetIdx=0 depicted in the table of  FIG. 1  is selected, and as the transform set TransformSetV of a primary vertical transform PTver (also referred to as primary vertical transform set), the transform set of the transform set identifier TransformSetIdx=2 depicted in the table of  FIG. 1  is selected. 
     Further, which one of the orthogonal transforms in the selected transform set TransformSet is to be applied to the primary horizontal transform is selected by a primary horizontal transform designation flag pt_hor_flag. Further, which one of the orthogonal transforms in the selected transform set TransformSet is to be applied to the primary vertical transform is selected by a primary vertical transform designation flag pt_hor_flag. For example, the primary horizontal transform PThor and the primary vertical transform PTver are derived from a definition table (LUT_TransformSetToTransformType) of transform sets depicted in  FIG. 1  taking the primary {horizontal, vertical} transform sets TransformSet{H, V} and the primary {horizontal, vertical} transform designation flags pt_{hor, ver}_flag as arguments as indicated by the following expressions (3) and (4), respectively. 
         PT hor=LUT_TransformSetTransformType [TransformSet H ][ pt _hor_flag]   (3)
 
         PT ver=LUT_TransformSetToTransformType [TrasnformSet V ][ pt _ver_flag]   (4)
 
     For example, in the case of the intra prediction mode number 9 (IntraMode==9), since the value of the transform set identifier of the primary horizontal transform set TransformSetH is 0, a primary horizontal transform is selected (designated) from within a transform set having the transform set identifier TransformSetIdx==0 on the transform set definition table LUT_TransformSetToTransformType of  FIG. 1 . In particular, in the case where the primary horizontal transform designation flag pt_hor_flag is 0, DT-VII is selected as the primary horizontal transform PThor, but in the case where the primary horizontal transform designation flag pt_hor_flag is 1, DCT-VIII is selected as the primary horizontal transform PThor. 
     Further, the primary transform identifier pt_idx is derived in accordance with the following expression (5) from the primary horizontal designation flag pt_hor_flag and the primary vertical transform designation flag pt_ver_flag. 
         pt _idx=( pt _ver_flag&lt;&lt;1)+ pt _hor_flag   (5)
 
     In particular, the upper 1 bit of the primary transform identifier pt_idx corresponds to a value of the primary vertical transform designation flag, and the lower 1 bit corresponds to a value of the primary horizontal transform designation flag. For a bin string of the derived primary transform identifier pt_idx, encoding is carried out by applying arithmetic encoding to generate a bit string. 
     In NPL 2, as orthogonal transforms configuring primary transform set, it is disclosed, that, in addition to the candidates for orthogonal transforms {DST-VII, DST-I, DCT-VIII} of NPL 1, Identity Transform (also called IDT or one-dimensional transform skip (1D Transform Skip)) of skipping DST-IV and a one-dimensional orthogonal transform and performing only scaling. Further, in NPL 2, a new transform set (transform set identifier TransformSetIdx==3) is added in  FIG. 1 . 
     Further, in the case of NPL 2, a definition of a transform set is determined on the basis of a transform block size and mode information. An example of the definitions of transform sets in 4×4/8×8/16×16/32×32 transforms in the case where the intra prediction mode number is 9 is depicted in a table of  FIG. 4 . For example, in the table depicted in  FIG. 4 , in the case of the transform block size 4×4, in the transform set of the transform set identifier TrasnformSetIdx=2, IDT is selected as the primary horizontal transform PThor, and DST-VII is selected as the primary vertical transform PTver. In particular, as represented by the expressions (6) and (7) given below, from the lookup table LUT_TransformSet, orthogonal transforms of the primary horizontal transform PThor and the primary vertical transform PTver are selected taking the intra prediction mode IntraMode as an argument for the first array, taking a value of a logarithm−2 of a transform block size as an argument for the second array, taking the transform set identifier TransformSetIdx as an argument for the third array and taking the horizontal direction H (=0) or the vertical direction V (=1) as an argument for the fourth array. 
         PT hor=LUT_TransfomSet[IntraMode(=9)][log 2TBSize−2(=0)][TransformSetIdx(=2)][ H (=0)]=IDT   (6)
 
         PT ver=LUT_TransfomSet[IntraMode(=9)][log 2TBSize−2(=0)][TransformSetIdx(=2)][ V (=1)]=DST-VII   (7)
 
     It is to be noted that, in NPL 2, the primary transform identifier pt_idx corresponds to the transform set identifier TransfromSetIdx as indicated by the following expression (8). 
         pt _idx=TransformSetIdx   (8)
 
     In NPL 2, there is a limit that the one-dimensional transform skip can be selected only in the case of a specific transform block size and a specific intra prediction mode number. Accordingly, in the case where it is better from a point of view of rate distortion to select one-dimensional transform skip of skipping an orthogonal transform in the horizontal or vertical direction rather than a two-dimensional orthogonal transform, since one-dimensional transform skip cannot be selected on the encoder side, there is the possibility that the encoding efficiency may be reduced. 
     Therefore, on the encoding side, execution of a primary horizontal transform that is a primary transform in the horizontal direction for residual data between an image and a prediction image is controlled on the basis of a value of a transform skip identifier and execution of a primary vertical transform that is a primary transform in the vertical direction for the residual data between the image and the prediction image is controlled on the basis of the value of the transform skip identifier, and the transform skip identifier is encoded. Meanwhile, on the decoding side, encoded data is decoded, and execution of an inverse primary vertical transform that is an inverse primary transform in the vertical direction for transform coefficient data transformed from image data is controlled on the basis of a value of a transform skip identifier obtained by the decoding of the encoded data and execution of an inverse primary horizontal transform that is an inverse primary transform in the horizontal direction for the coefficient data converted from the image data is controlled on the basis of the value of the transform skip identifier. 
     By such control as just described, in the case where it is desirable to skip a one-dimensional transform in the horizontal direction or the vertical direction, it is possible to suppress reduction of the processing amount of the (inverse) primary transform and reduction of the energy compaction to enhance the encoding efficiency. 
     &lt;Block Segmentation&gt; 
     In an old-fashioned image encoding method such as MPEG2 (Moving Picture Experts Group 2 (ISO/IEC 13818-2)) or MPEG-4 Part 10 (Advanced Video Coding, hereinafter referred to as AVC), an encoding process is executed in a processing unit called macro block. The macro block is a block having a uniform size of 16×16 pixels. In contrast, in HEVC (High Efficiency Video Coding), an encoding process is executed in a processing unit (encoding unit) called CU (Coding Unit). A CU is a block having a variable size, which is formed by recursively segmenting an LCU (Largest Coding Unit) that is a maximum encoding unit. The maximum size of a CU that can be selected is 64×64 pixels. The minimum size of a CU that can be selected is 8×8 pixels. A CU of the minimum size is called SCU (Smallest Coding Unit). It is to be noted that the maximum size of a CU is not limited to 64×64 pixels but may be a greater block size such as 128×128 pixels, 256×256 pixels or the like. 
     As a result of adoption of a CU having a variable size in this manner, according to HEVC, it is possible to adaptively adjust the picture quality and the encoding efficiency in response to the substance of an image. A prediction process for prediction encoding is executed in a processing unit (prediction unit) called PU (Prediction Unit). A PU is formed by segmenting a CU in one of several segmentation patterns. Further, a PU includes a processing unit (prediction block) called PB (Prediction Block) for each of the luminance (Y) and the color differences (Cb and Cr). Furthermore, an orthogonal transform process is executed in a processing unit (transform unit) called TU (Transform Unit). A TU is formed by segmenting a CU or a PU to a certain depth. Further, a TU includes a processing unit (transform block) called TB (Transform Block) for each of the luminance (Y) and the color differences (Cb and Cr). 
     &lt;Recursive Block Segmentation&gt; 
       FIG. 6  is an explanatory view illustrating an overview of recursive block segmentation regarding a CU in HEVC. The block segmentation of a CU is performed by recursively repeating segmentation of one block into four (=2×2) sub blocks, and as a result, a tree structure in the form of a quad tree (Quad-Tree) is formed. The entirety of one quad tree is called CTB (Coding Tree Block), and a logical unit corresponding to the CTB is called CTU (Coding Tree Unit). 
     At an upper portion in  FIG. 6 , C01 that is a CU having a size of 64×64 pixels is depicted as an example. The depth of segmentation of C01 is equal to zero. This signifies that C01 is the root of a CTU and corresponds to an LCU. The LCU size can be designated by a parameter that is encoded in an SPS (Sequence Parameter Set) or a PPS (Picture Parameter Set). C02 that is a CU is one of four CUs segmented from C01 and has a size of 32×32 pixels. The depth of segmentation of C02 is equal to 1. C03 that is a CU is one of four CUs segmented from C02 and has a size of 16×16 pixels. The depth of segmentation of C03 is equal to 2. C04 that is a CU is one of four CUs segmented from C03 and has a size of 8×8 pixels. The depth of segmentation of C04 is equal to 3. In this manner, a CU is formed by recursively segmenting an image to be encoded. The depth of segmentation is variable. For example, to a flag image region like the blue sky, a CU of a comparatively great size (namely, of a small depth) can be set. On the other hand, to a steep image region including many edges, a CU of a comparatively small size (namely, of a great depth) can be set. Then, each of such set CUs becomes a processing unit in an encoding process. 
     &lt;Setting of PU to CU&gt; 
     A PU is a processing unit in a prediction process including intra prediction and inter production. A PU is formed by segmenting a CU in one of several segmentation patterns.  FIG. 7  is an explanatory view illustrating setting of a PU to a CU depicted in  FIG. 6 . In a right region in  FIG. 7 , eight segmentation patterns of 2N×2N, 2N×N, N×2N, N×N, 2N×nU, 2N×nD, nL×2N and nR×2N are depicted. In intra prediction, the two segmentation patterns of 2N×2N and N×N can be selected from among the eight segmentation patterns (N×N can be selected only in the SCU). In contrast, in inter prediction, all of the eight segmentation patterns can be selected in the case where asymmetrical motion segmentation is enabled. 
     &lt;Setting of TU to CU&gt; 
     A TU is a processing unit of an orthogonal transform process. A TU is formed by segmenting a CU (in regard to an intra CU, each PU in the CU) to a certain depth.  FIG. 8  is an explanatory view illustrating setting of a TU to a CU depicted in  FIG. 7 . In a right region in  FIG. 8 , one or more TUs that can be set to C02 are depicted. For example, T01 that is a TU has a size of 32×32 pixels, and the depth of the TU segmentation is equal to zero. T02 that is a TU has a size of 16×16 pixels, and the depth of the TU segmentation is equal to 1. T03 that is a TU has a size of 8×8 and the depth of the TU segmentation is equal to 2. 
     What block segmentation is to be performed in order to set such a block as a CU, a PU or a TU described above is determined typically on the basis of comparison in cost that affects the encoding efficiency. An encoder compares the cost, for example, between one CU of 2M×2M pixels and four CUs of M×M pixels, and if the setting of four CUs of M×M pixels indicates a higher encoding efficiency, then the encoder determines to segment a CU of 2M×2M into four CUs of M×M segments. 
     &lt;Scanning Order of CUs and PUs&gt; 
     When an image is to be encoded, a CTB (or an LCU) set in a lattice-like pattern in the image (or in a slice or a tile) is scanned in a raster scan order. In one CTB, CUs are scanned so as to follow the quad tree from the left to the right and from the top to the bottom. When a current block is to be processed, information of the upper and left adjacent blocks is utilized as input information.  FIG. 9  is an explanatory view illustrating a scanning order of CUs and PUs. At a left upper portion in  FIG. 9 , C10, C11, C12 and C13 that are four CUs that can be included in one CTB are depicted. A numeral in a framework of each CU represents an order number of the process. The encoding process is executed in an order of C10 that is the left upper CU, C11 of the right upper CU, C12 of the left lower CU and C13 of the right lower CU. At a right portion in  FIG. 9 , one or more PUs for inter prediction capable of being set to C11 that is a CU are depicted. At a lower portion of  FIG. 9 , one or more PUs for intra prediction capable of being set to C12 that is a CU are depicted. As indicated by numerals in frameworks of the PUs, also the PUs are scanned so as to follow from the left to the right and from the top to the bottom. 
     In the following description, description is sometimes given using a “block” as a partial region or a processing unit of an image (picture) (the “block” is not a block of a processing section). The “block” in this case indicates an arbitrary partial region in the picture, and the size, shape, characteristic or the like of it is not restricted. In other words, it is assumed that the “block” in this case includes an arbitrary partial region (processing unit) such as, for example, a TB, a TU, a PB, a PU, an SCU, a CU, an LCU (CTB), a sub block, a macro block, a tile, a slice or the like. 
     &lt;Image Decoding Apparatus&gt; 
       FIG. 10  is a block diagram depicting an example of a configuration of an image decoding apparatus that is a form of an image processing apparatus to which the present technology is applied. An image decoding apparatus  100  depicted in  FIG. 10  is an apparatus that decodes encoded data encoded from a prediction residual between an image and a prediction image of the image as in AVC or HEVC. For example, the image decoding apparatus  100  incorporates the technology proposed by HEVC or the technology proposed by JVET (Joint Video Exploration Team). 
     Referring to  FIG. 10 , the image decoding apparatus  100  includes a decoding section  111 , a dequantization section  112 , an inverse transform section  113 , an arithmetic operation section  114 , a frame memory  115  and a prediction section  116 . It is to be noted that the prediction section  116  includes an intra prediction section and an inter prediction section not depicted. The image decoding apparatus  100  is an apparatus for generating a moving image #2 by decoding encoded data #1 (bit stream). 
     The decoding section  111  receives encoded data #1 as an input thereto and variable length decodes syntax values of syntax elements from a bit string of the encoded data #1 in accordance with a definition of a syntax table. Furthermore, the syntax elements include such information as header information Hinfo, prediction mode information Pinfo, transform information Tinfo, residual information Rinfo and so forth. 
     The header information Hinfo such as VPS/SPS/PPS/slice header SH includes information that prescribes an image size (horizontal width PicWidth, vertical width PicHeight), a bit depth (luminance bitDepthY, color difference bitDepthC), a maximum value MaxCUsize/minimum value MinCUSize of the CU size, a maximum depth MaxQTDepth/minimum depth MinQTDepth of quad-tree segmentation (also referred to as Quad-tree segmentation), a maximum depth MaxBTDepth/minimum depth MinBTDepth of binary tree segmentation (Binary-tree segmentation), a maximum value MaxTSSize of the transform skip block (referred to also as maximum transform skip block size), an on/off flag (also referred to as validity flag) of each encoding tool and so forth. 
     For example, as the on/off flags for encoding tools included in the header information Hinfo, on/off flags relating to transform and quantization processes indicated below are available. It is to be noted that the on/off flag for each encoding tool can be interpreted also as a flag indicative of whether or not syntax relating to the encoding tool exists in encoded data. Further, in the case where the value of the on/off flag is 1 (true), this indicates that the encoding tool is usable, but in the case where the value of the on/off flag is 0 (false), this indicates that the encoding tool is not usable. It is to be noted that the interpretations of the flag value may be reversed. 
     The adaptive primary transform validity flag apt_enabled_flag (also referred to as adaptive_primary_transform_enabled_flag, adaptive_pt_enabled_flag, or amt_enabled_flag) is a flag indicative of whether, as one of transform processes and inverse processes to them, an encoding tool that can select an adaptive primary transform is usable. 
     The secondary transform validity flag st_enabled_flag is a flag indicative of whether or not an encoding tool for performing a secondary transform/inverse secondary transform as one of transform processes and inverse processes is usable. 
     The transform quantization bypass validity flag transquant_bypass_enabled_flag is a flag indicative of whether or not an encoding tool for skipping, as one of transforms and quantization and inverse processes to them, a transform, quantization/dequantization and an inverse transform is usable. 
     The transform skip flag validity flag ts_enabled_flag is a flag indicative of whether or not, as one of transform processes and inverse processes to the transform processes, two-dimensional transform skip or one-dimensional transform skip is usable. The two-dimensional transform skip is an encoding tool for skipping orthogonal transforms and inverse processes to the orthogonal transforms (inverse orthogonal transforms) including primary transforms and secondary transforms. Meanwhile, the one-dimensional transform skip is an encoding tool for skipping, from among primary transforms, a primary transform in the horizontal direction or the vertical direction and an inverse transform corresponding to the primary transform (inverse primary transform) as well as a secondary transform and an inverse secondary transform to the secondary transforms. 
     The prediction mode information Pinfo further includes a PU size (prediction block size) PUSize of a processing target PU, intra prediction mode information IPinfo (for example, prev_intra_luma_pred_flag, mpm_idx, and rem_intra_pred_mode in JCTVC-W1005, 7.3.8.5 Coding Unit syntax), motion prediction information MVinfo (for example, refer to JCTVC-W1005, 7.3.8.6 Prediction Unit Syntax, merge_idx, merge_flag, inter_pred_idc, ref_idx_LX, mvp_lX_flag, X={0,1}, and mvd) and so forth. 
     Meanwhile, the transform information Tinfo includes syntax of, for example, a horizontal width size TBWidth and a vertical width TBHeight of a processing target transform block, a transform quantization bypass flag transquant_bypass_flag indicative of whether or not processes for (inverse) transforms and (de) quantization are to be skipped, a transform skip identifier ts_idx that designates various transform skip mods such as two-dimensional transform skip or one-dimensional transform skip, an adaptive primary transform flag apt_flag indicative of whether an adaptive primary transform is to be applied to a target TU, a primary transform identifier pt_idx indicative of which one of (inverse) primary transforms is to be applied for (inverse) primary transforms in each of the vertical direction and the horizontal direction, a secondary transform identifier st_sdx (also referred to as dnsst_idx, nsst_idx, or rot_idx) indicative of which one of (inverse) secondary transforms is to be applied, a scan identifier scanIdx, a quantization parameter qp, a quantization matrix scaling_matrix and so forth. It is to be noted that, in place of the horizontal width size TBWidth and the vertical width TBHeight of a processing target transform block, logarithms log 2TBWidth and log 2TBHeight of TBWidth and TBHeight with the base 2 may be included, respectively, in the transform information Tinfo. 
     The residual information Rinfo includes, for example, a last non-zero coefficient X coordinate (last_sig_coeff_x_pos), a last non-zero coefficient Y coordinate (last_sig_coeff_y_pos), a sub block non-zero coefficient presence/absence flag (coded_sub_block_flag), a non-zero coefficient presence/absence flag (sig_coeff_flag), a flag (gr1_flag) (also referred to as GR1_flag) that indicates whether the level of a non-zero coefficient is greater than 1, a flag (gr2_flag) (also referred to as GR2_flag) that indicates whether the level of a non-zero coefficient is greater than 2, a sign (sign_flag) (also referred to as sign code) that represents whether the non-zero coefficient is in the positive or in the negative, a remaining level of a non-zero coefficient (coeff_abs_level_remaining) (also referred to as non-zero coefficient remaining level) and so forth. 
     The decoding section  111  refers to the residual information Rinfo to derive quantization transform coefficient levels level of coefficient positions in transform blocks. The decoding section  111  supplies the prediction mode information Pinfo, quantization transform coefficient levels level and transform information Tinfo obtained by the decoding to the associated blocks. For example, the decoding section  111  supplies the prediction mode information Pinfo to the prediction section  24 , supplies the quantization transform coefficient levels level to the dequantization section  22  and supplies the transform information Tinfo to the inverse transform section  113  and the dequantization section  112 . 
     The dequantization section  112  receives the transform information Tinfo and the quantization transform coefficient levels level as inputs thereto and scales (dequantizes) the values of the quantization transform coefficient levels level on the basis of the transform information Tinfo and outputs transform coefficients Coeff_IQ after the dequantization to the inverse transform section  113 . 
     The inverse transform section  113  receives the transform coefficients Coeff_IQ and the transform information Tinfo as inputs thereto and applies an inverse transform to the transform coefficients Coeff_IQ to derive a prediction residual D′, and outputs the prediction residual D′ to the arithmetic operation section  114 . Details of the inverse transform section  113  are hereinafter described. 
     The arithmetic operation section  114  receives the prediction residual D′ and prediction images P that are supplied from the prediction section  116  as inputs thereto, adds the prediction residual D′ and the prediction image P (prediction signal) corresponding to the prediction residuals D′ to derive locally decoded images Rec as indicated by an expression (9) given below, and supplies the locally decoded images Rec to the frame memory  115  or to the outside of the image decoding apparatus  100 . 
       Rec= D′+P    (9)
 
     The frame memory  115  receives the locally decoded images Rec supplied from the arithmetic operation section  114  as an input thereto, and re-constructs a decoded image for each picture unit and stores the decoded images into the buffer in the frame memory  115 . The frame memory  115  reads out a decoded image designated by the prediction mode information Pinfo of the prediction section  116  as a reference image from the buffer and supplies the reference image to the prediction section  116 . Further, the frame memory  115  may store header information Hinfo, prediction mode information Pinfo, transform information Tinfo and so forth relating to the generation of the decoded image into the buffer in the frame memory  115 . 
     The prediction section  116  receives the prediction mode information Pinfo as an input thereto, and generates a prediction image P by a prediction method designated by the prediction mode information Pinfo using a decoded image stored in the frame memory  115  and designated by prediction mode information PInfo as a reference image and outputs the generated prediction image P to the arithmetic operation section  114 . 
       FIG. 11  is a block diagram depicting a principal configuration example of the inverse transform section  113  provided in the image decoding apparatus  100  of  FIG. 10 . As depicted in  FIG. 11 , the inverse transform section  113  includes a switch  121 , an inverse secondary transform section  122  and an inverse primary transform section  123 . 
     The switch  121  receives transform coefficients Coeff_IQ and a transform skip identifier ts_idx as inputs thereto. In the case where the value of the transform skip identifier ts_idx is NO_TS(=0) or 1D_H_TS(=2) or else 1D_V_TS(=3) (in the case where it is indicated that transform skip is not to be applied or one-dimensional skip is to be applied to one of the horizontal and vertical directions), the switch  121  outputs the transform coefficients Coeff_IQ to the inverse secondary transform section  122 . On the other hand, in the case where the value of the transform skip flag ts_idx is 2D_TS(=1) (in the case where it is indicated that two-dimensional transform skip is to be applied), the switch  121  skips the inverse secondary transform section  122  and the inverse primary transform section  123  and outputs the transform coefficients Coeff_IQ as a prediction residual D′. 
     For example, in the case where the transform skip identifier ts_idx is 2D TS(=1) and the transform coefficients Coeff_IQ to be inputted to the inverse transform section  113  is a 4×4 matrix Coeff_IQ=[[255, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]], the switch  121  outputs the transform coefficients Coeff_IQ as a prediction residual D′. In particular, the prediction residual D′ becomes D′=[[255, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]. Accordingly, an inverse secondary transform and an inverse primary transform can be skipped. Especially, it is possible to perform, for a sparse residual signal in which the number of non-zero coefficients is small and to which it is desirable to apply two-dimensional transform skip, an inverse transform process that achieves reduction of the processing amount of an inverse transform and enhancement of the encoding efficiency. 
     The inverse secondary transform section  122  receives a secondary transform identifier st_idx, a scan identifier scanIdx indicative of a scanning method for transform coefficients and transform coefficients Coeff_IQ as inputs thereto and derives and supplies transform coefficients Coeff_IS after the inverse secondary transform (also referred to as primary transform coefficients) to the inverse primary transform section  123 . More particularly, in the case where the secondary transform identifier st_idx indicates that an inverse secondary transform is to be applied (st_idx&gt;0), the inverse secondary transform section  122  executes a process of an inverse secondary transform corresponding to the secondary transform identifier st_idx for the transform coefficients Coeff_IQ, and outputs transform coefficients Coeff_IS after the inverse secondary transform. In the case where the secondary transform identifier st_idx indicates that an inverse secondary transform is not to be applied (st_idx==0), the inverse secondary transform is skipped and the transform coefficients Coeff_IQ are outputted as the transform coefficients Coeff_IS after the inverse secondary transform. 
     The inverse primary transform section  123  receives a primary horizontal transform designation flag pt_hor_flag, a primary vertical transform designation flag pt_ver_flag, prediction mode information PInfo, a transform skip identifier ts_idx and transform coefficients Coeff_IS after an inverse secondary transform as inputs thereto. The inverse primary transform section  123  selects a matrix IPThor (=PThor T ) of an inverse primary horizontal transform and a matrix IPTver (=PTver T ) of inverse primary vertical transform designated by the prediction mode information PInfo, the transform skip identifier and the primary horizontal transform designation flags pt_hor_flag and primary vertical transform designation flag pt_ver_flag, performs an inverse primary horizontal transform and an inverse primary vertical transform for the transform coefficients Coeff_IS after the inverse secondary (also referred to as primary transform coefficients Coeff_P) with the selected matrices of the inverse primary transforms in the directions to derive prediction residuals D′ after the inverse primary transform, and outputs the prediction residuals D′. 
     As depicted in  FIG. 11 , the inverse primary transform section  123  includes an inverse primary transform selection section  131 , a switch  132 , an inverse primary vertical transform section  133 , another switch  134  and an inverse primary horizontal transform section  135 . 
     The inverse primary transform selection section  131  receives a primary horizontal transform designation flag pt_hor_flag, a primary vertical transform designation flag pt_ver_flag, prediction mode information PInfo and a transform skip identifier ts_idx as inputs thereto, reads out a matrix IPThor (=PThor T ) of an inverse primary horizontal transform and a matrix IPTver (=PTver T ) of an inverse primary vertical transform designated by the prediction mode information PInfo, the transform skip identifier ts_idx, and a primary horizontal transform designation flag pt_hor_flag and a primary vertical transform designation flag pt_ver_flag from an internal memory (not depicted) of the inverse primary transform selection section  131 , and supplies them to the inverse primary horizontal transform section  135  and the inverse primary vertical transform section  133 . 
     More particularly, in the case where the adaptive primary transform flag apt_flag (also referred to as CU primary transform flag cu_pt_flag) is 1 (true), the inverse primary transform selection section  131  selects, for example, from among three transform sets TransformSet of the transform set identifier TransfromSetIdx=0, . . . , 2 depicted in the table of  FIG. 1 , a transform set TransformSet, which includes orthogonal transforms that become candidates for a primary transform, for each of the horizontal direction and the vertical direction on the basis of the correspondence table (intra prediction mode information) between mode information and transform sets depicted in  FIG. 3 . It is to be noted that, in FIG.  3 , the intra prediction mode number 35 may be treated as a mode indicative of an inter prediction (Inter) or of an intra block copy (IBC: Intra Block Copy). 
     For example, the selection is carried out such that, for each of transform sets TransformSet{H, V}, a transform set identifier TransformSetIdx for designating a corresponding transform set TransformSet as represented by the following conditional expression (10). 
       if (apt_flag) 
       TransformSetH=LUT_IntraModeToTransformSet [IntraMode][ H (=0)] 
       TransformSet V =LUT_IntraModeToTransformSet [IntraMode][ V (=1)] 
       else//apt_flag==0 
       TransformSet H =TransformSet V =3 (predetermined value)   (10)
 
     Here, TransformSetH indicates a transform set of primary horizontal transforms PThor while TransformSetV indicates a transform set of primary vertical transforms PTver, and a lookup table LUT_IntraModeToTransformSet indicates the correspondence table of  FIG. 3 . The first array of the lookup table LUT_IntraModeToTransformSet[ ][ ] takes the intra prediction mode IntraMode as an argument, and the second array takes {H=0, V=1} as an argument. For example, in the case of the intra prediction mode number 9 (IntraMode==9), as the transform set TransformSetH of a primary horizontal transform PThor (also referred to as primary horizontal transform set), the transform set of the transform set identifier TransformSetIdx=0 depicted in the table of  FIG. 1  is selected, and as the transform set TransformSetV of a primary vertical transform PTver (also referred to as primary vertical transform set), the transform set of the transform set identifier TransformSetIdx=2 depicted in  FIG. 1  is selected. On the other hand, in the case where the adaptive primary transform flag amt_flag is 0 (false), the inverse primary transform selection section  131  sets the transform set TransformSet of the transform set identifier TransfromSetIdx=3 (predetermined value) depicted in the table of  FIG. 1  as a transform set including orthogonal transforms that become candidates for a primary transform. 
     Furthermore, the inverse primary transform selection section  131  selects, for each horizontal/vertical direction, orthogonal transforms to be used for an inverse primary transform from the selected transform set TransformSet depending upon the primary horizontal transform designation flag pt_hor_flag and the primary vertical transform designation flag pt_ver_flag, respectively. 
     For example, as in the conditional expression (11) given below, in the case where the primary horizontal transform designation flag pt_hor_flag is −1, a matrix IDT corresponding to one-dimensional transform skip is selected, but in the case where the primary horizontal transform designation flag pt_hor_flag is any other than −1, an orthogonal transform is derived from the definition table (LUT_TransformSetToTransformType) of the transform sets depicted in  FIG. 1  taking the primary horizontal transform set TransformSetH and the primary horizontal transform designation flag pt_hor_flag as arguments. 
       if ( pt _hor_flag !=−1)
 
         IPT hor=LUT_TransformSetToTransformType [TransformSet H ][ pt _hor_flag] 
       else 
         IPT hor=IDT   (11)
 
     For example, if an inverse primary horizontal transform in the case of the intra prediction mode number 9 (IntraMode==9) is taken as an example, then since the value of the transform set identifiers of the primary horizontal transform set TransformSetH is 0, an inverse primary horizontal transform is selected from within the transform set whose transform set identifier TransformSetIdx==0 on the transform set definition table LUT_TransformSetToTransformType of  FIG. 1 . In particular, in the case where the primary horizontal transform designation flag pt_hor_flag==−1, IDT is selected as the inverse primary horizontal transform IPThor; in the case where the primary horizontal transform designation flag pt_hor_flag==0, DST-VII is selected as the inverse primary horizontal transform IPThor; and in the case where the primary horizontal transform designation flag pt_hor_flag is 1, DCT-VIII is selected as the inverse primary horizontal transform IPThor. 
     Similarly, also in regard to an inverse primary vertical transform, as indicated by the conditional expression (12) given below, in the case where the primary vertical transform designation flag pt_ver_flag is −1, the matrix IDT (unit matrix) corresponding to one-dimensional transform skip, and in the case where the primary vertical transform designation flag pt_ver_flag is any other than −1, an orthogonal transform is derived from within the definition table (LUT_TransformSetToTransformType) of transform sets depicted in  FIG. 1  taking the primary vertical transform set TransformSetV and the primary vertical transform designation flag pt_ver_flag as arguments. 
     
       
         
           
               
               
               
             
               
                   
                   
               
             
            
               
                   
                   
                  if (pt_ver_flag != −1) 
               
               
                   
                   
                  IPTver = LUT_TransformSetToTransformType 
               
               
                   
                   
                 [ TrasnformSetV ] [ pt_ver_flag ] 
               
               
                   
                   
                  else 
               
               
                   
                   
                  IPTver = IDT 
               
               
                   
                   
                 . . . (12) 
               
               
                   
                   
               
            
           
         
       
     
     It is to be noted that, although, in the expressions (11) and (12) given above, in the case of pt_{hor, ver} flag==−1, IDT is set by conditional branching, IDT may be set referring to the lookup table LUT_TransformSetToTransformType[ ][ ]. In this case, it is sufficient if the value of the element of the lookup table LUT_TransformSetToTransformType[0 . . . 3][−1] is set to IDT. It is to be noted that, while, in this example, pt_{hor, ver} flag==−1 is a value representative of one-dimensional transform skip, the value can be freely changed as far as practicable. Further, the combination in value of a transform set identifier and pt_{hor, ver}_flag in  FIG. 1  may be changed freely as far as practicable. For example, in the case where pt_{hor, ver}_flag==1, DST-VII may be set irrespective of the value of the transform set identifier, and in the case where pt_{hor, ver} flag==0, DCT-VIII, DST-I, DCT-V and DST-VII may be set in order of the transform set identifiers 0 to 3, respectively. Further, the order of the transform set identifiers 0 to 2 may be changed freely as far as practicable. Further, the inverse primary transform selection section  131  may select, in place of the intra prediction mode information IPinfo, an inverse primary horizontal transform IPThor and an inverse primary vertical transform IPTver in response to motion prediction information MVinfo and the primary horizontal transform designation flag pt_hor_flag and primary vertical transform designation flag pt_ver_flag. 
     Thereafter, the inverse primary transform selection section  131  reads out matrices of orthogonal transforms corresponding to the inverse primary horizontal transform IPThor and the inverse primary vertical transform IPTver from the buffer (not depicted) held by the inverse primary transform section  123  and supplies the matrices corresponding to the inverse primary vertical transform and the inverse primary horizontal transform. 
     The switch  132  receives transform coefficients Coeff_IS after the inverse secondary transform (also referred to primary transform coefficients Coeff_P) and a transform skip identifier ts_idx as inputs thereto. In the case where the value of the transform skip identifier ts_idx is 1D_V_TS(=3), namely, indicates that a one-dimensional transform in the vertical direction is to be skipped (ts_idx==1D_V_TS) (to skip the (inverse) primary vertical transform), the switch  132  skips processing of the inverse primary vertical transform section  133  and outputs the primary transform coefficient Coeff_IS as a transform coefficient Coeff_IPver after the inverse primary vertical transform. On the other hand, in the case where the value of the transform skip identifier ts_idx is any other than 1D_V_TS(=3) (ts_idx !=1D_V_TS) (indicates that the (inverse) primary vertical transform is not to be skipped), the switch  132  outputs the primary transform coefficient Coeff_IS. 
     The inverse primary vertical transform section  133  receives, for each transform block, a transform coefficient Coeff_IS after the inverse secondary transform and a matrix for an inverse primary vertical transform IPTver as inputs thereto, performs matrix operation represented by the expression (13) given below and outputs a result of the matrix operation as a transform coefficient Coeff_IPver after the inverse primary vertical transform. It is to be noted that the inverse primary vertical transform IPTver is an inverse transform to the primary vertical transform PTver having a transformation basis as a column vector and is represented by a matrix PTverr transposed from the primary vertical transform PTver. 
     
       
         
           
             
               
                 
                   Coeff_IPver 
                   = 
                   
                     
                       
                         ( 
                         
                           IPTver 
                           · 
                           Coeff_IS 
                         
                         ) 
                       
                        
                       
                         s 
                          
                         
                             
                         
                          
                         1 
                       
                     
                     = 
                     
                       
                         ( 
                         
                           
                             PTver 
                             T 
                           
                           · 
                           Coeff_IS 
                         
                         ) 
                       
                        
                       
                         s 
                          
                         
                             
                         
                          
                         1 
                       
                     
                   
                 
               
               
                 
                   ( 
                   13 
                   ) 
                 
               
             
           
         
       
     
     Here, the operator “·” represents a matrix product (inner product), the operator “T” represents an operation for the transposed matrix, and the operator “&gt;&gt;” represents an operation for performing right shift operation for the elements. According to the expression (13), each value of the transform coefficient Coeff_IPver is obtained by performing, for each element of the matrix product of the transform coefficient Coeff_IS and the transposed matrix PTver T  (=IPTver) after the primary vertical transform, right shift operation with a predetermined scaling parameter s1. It is to be noted that the scaling parameter s1 is used to normalize a matrix operation result of IPTver·Coeff_IS so as to fit in the bit depth of an intermediate buffer. The value of the scaling parameter s1 is determined from the bit depth BitDepthbuff of the intermediate buffer and the worst case MaxBitDepth (IPTver*Coeff_IS) of the bit depth of the matrix operation of IPTver*Coeff_IS, for example, as indicated by the following expression (14). 
         s 1=max(0,MaxBitDepth(IPTver*Coeff_ IS )−BitDepthbuff)   (14)
 
     For example, in the case where the value range of IPTver*Coeff_IS is −2**22 to 2**22−1, namely, in the case where the bit depth of MaxBitDepth(IPTver*Coeff_IS) is 23 bits and the value range of a value that can be stored into the intermediate buffer is −2**15 to 2**15−1, namely, the bit depth of the intermediate buffer is 16 bits, the scaling parameter s1 becomes s1=7 bits (=max(0, 23-16)=max(0, 7)). 
     On the other hand, in the case where the bit depth of MaxBitDepth(IPTver·Coeff_IS) is 23 bits and the value range of the value that can be stored into the intermediate buffer is −2**31 to 2**31−1, namely, in the case where the bit depth of the intermediate buffer is 32 bits, the scaling parameter s1 becomes s1=0 bit (=max(0, 23−32)=max(0, −9)). That the scaling parameter s1 is 0 represents that, since the value of the bit depth of the intermediate buffer is sufficiently high, even if element values of the matrix product of IPTver*Coeff_IS are not normalized, they can be stored into the intermediate buffer without overflowing. It is to be noted that the scaling parameter s1 may be a fixed value determined in advance assuming that the value range of IPTver*Coeff_IS is known. 
     Further, in order to reduce a clip error by right shift operation of the expression (13) described hereinabove, after the matrix product, a predetermined offset value o1 may be added for each element. At this time, the offset value o1 is represented by the following expression (15) using the scaling parameter s1. 
         o 1=( s 1&gt;0?1&lt;&lt;( s 1−1):0)   (15)
 
     The switch  134  receives a transform coefficient Coeff_IPver after the inverse primary vertical transform and a transform skip identifier ts_idx. In the case where the value of the transform skip identifier ts_idx is 1D_H_TS(=2), namely, indicates that a one-dimensional transform in the horizontal direction (ts_idx==1D_H_TS) (to skip a (inverse) primary horizontal transform), the switch  134  skips the inverse primary horizontal transform section  135  and outputs the transform coefficient Coeff_IPver after the inverse primary vertical transform as a prediction residual D′ to the outside. On the other hand, in the case where the value of the transform skip identifier ts_idx is any other than 1D_H_TS(=2) (ts_idx !=1D_H_TS) (in the case where it is indicated that a (inverse) primary horizontal transform is not to be skipped), the switch  134  outputs the transform coefficient Coeff_IPver after the inverse primary vertical transform to the inverse primary horizontal transform section  135 . 
     The inverse primary horizontal transform section  135  receives, for each transform block, transform coefficients Coeff_IPver after the inverse primary vertical transform and a matrix of an inverse primary horizontal transform IPThor as inputs thereto, performs matrix operation as indicated by the expression (16) given below, and outputs a result of the matrix operation as a prediction residual D′. It is to be noted that the inverse primary horizontal transform IPThor is an inverse transform of a primary horizontal transform PThor that has the transformation basis as a row vector, and is represented by a matrix PThor T  transposed from the primary horizontal transform PThor. 
     
       
         
           
             
               
                 
                   
                     D 
                     ′ 
                   
                   = 
                   
                     
                       
                         ( 
                         
                           Coeff_IPver 
                           · 
                           IPThor 
                         
                         ) 
                       
                        
                       
                         s 
                          
                         
                             
                         
                          
                         2 
                       
                     
                     = 
                     
                       
                         ( 
                         
                           Coeff_IPver 
                           · 
                           
                             PThor 
                             T 
                           
                         
                         ) 
                       
                        
                       
                         s 
                          
                         
                             
                         
                          
                         2 
                       
                     
                   
                 
               
               
                 
                   ( 
                   16 
                   ) 
                 
               
             
           
         
       
     
     Here the operator “·” represents a matrix product (inner product), the operator “T” represents an operation for a transposed matrix, and the operator “&gt;&gt;” represents an operation for performing right shift operation for each element. According to the expression (16), each value of the prediction residual D′ is obtained by performing, for each element of the matrix product of the transform coefficients Coeff_IPver and the transposed matrix PThor T  (=IPTver) of the primary horizontal transform, right shift operation with a predetermined scaling parameter s2. It is to be noted that the scaling parameter s2 is used to normalize a matrix operation result of Coeff_IPver·IPThor so as to fit in a desire bit depth. The value of the scaling parameter s2 is determined from a desired bit depth BitDepthout and the worst case MaxBitDepth(Coeff_IPver-IPThor) of the bit depth of the matrix operation of Coeff_IPver*IPThor in accordance with the following expression (17). 
         s 2=max(0,MaxBitDepth(Coeff_ IP ver· IPT hor)−BitDpethout)   (17)
 
     For example, in the case where the value range of Coeff_IPver·IPThor is −2**22 to 2**22−1, namely, in the case where the bit depth of MaxBitDepth(Coeff_IPver*IPThor) is 23 bits and the value range of the value that can be taken with a desired bit depth is −2*+15 to 2**15-1, namely, in the case where the desired bit depth is 16 bits, the scaling parameter s2 becomes s2=7 bits (=max(0, 23-16)=max(0, 7)). 
     Further, in the case where the bit depth of MaxBitDepth(Coeff_IPver*IPThor) is 23 bits and the value range of the value that can be taken with a desired bit depth is −2**31 to 2**31−1, namely, in the case where the desired bit depth is 32 bits, the scaling parameter s2 becomes s2=0 bit (=max(0, 23-32)=max(0, −9)). Since that the scaling parameter s2 is 0 represents that, since the value of the desired bit depth is sufficiently high, even if the pixel values of the matrix product of Coeff_IPver*IPThor are not normalized, the pixel values fit in the desired bit depth. 
     It is to be noted that, in order to reduce a clip error by right shift operation of the expression (16) described hereinabove, after the matrix product, a predetermined offset value o2 may be added for each element. At this time, the offset value o2 is represented by the following expression (18) using the scaling parameter s2. 
         o 2=( s 2&gt;0?1&lt;&lt;( s 2−1):0)   (18)
 
     As described above, the inverse primary transform section  123  can perform, for a residual signal in regard to which it is desirable to skip a one-dimensional transform in the horizontal direction or the vertical direction, an inverse primary transform process that decreases the processing amount of an inverse primary transform and prevents decrease of energy compaction to enhance the encoding efficiency. 
     Especially, for a residual signal having a characteristic of a step edge in which the continuity of a signal changes rapidly in the horizontal direction, by skipping a one-dimensional transform in the horizontal direction and performing a one-dimensional transform in the vertical direction, non-zero coefficients can be concentrated efficiently on a low frequency region of frequency components in the vertical direction in comparison with those in the case where a two-dimensional orthogonal transform is performed. In particular, since energy compaction can be increased, enhancement of the encoding efficiency can be implemented. 
     Similarly, for a residual signal having a characteristic of a step edge in which the continuity of a signal changes rapidly in the vertical direction, by skipping a one-dimensional transform in the vertical direction and performing a one-dimensional transform in the horizontal direction, non-zero coefficients can be concentrated efficiently on decrease of frequency components in the horizontal direction in comparison with those in an alternative case in which a two-dimensional orthogonal transform is performed. In particular, since energy compaction can be increased, enhancement of the encoding efficiency can be implemented. 
     &lt;Flow of Image Decoding Process&gt; 
     Now, a flow of processes executed by such an image decoding apparatus  100  described above is described. First, a flow of an image decoding process is described with reference to a flow chart of  FIG. 12 . 
     After an image decoding process is started, at step S 101 , the decoding section  111  decodes a bit stream (encoded data) supplied to the image decoding apparatus  100  and acquires information such as header information Hinfo, prediction mode information Pinfo, transform information Tinfo, residual information Rinfo, quantization transform coefficient levels level and so forth. 
     At step S 102 , the dequantization section  112  dequantizes the quantization transform coefficient levels level obtained by the process at step S 101  to derive transform coefficients Coeff_IQ. This dequantization is an inverse process to the quantization performed in an image encoding process hereinafter described and is a process similar to dequantization performed in the image encoding process. 
     At step S 103 , the inverse transform section  113  inversely transforms the transform coefficients Coeff_IQ obtained by the process at step S 102  to derive a prediction residual D′. This inverse transform is an inverse process to a transform process performed in the image encoding process hereinafter described and is a process similar to that of an inverse transform performed in the image encoding process. 
     At step S 104 , the prediction section  116  performs prediction in a prediction mode same as that in prediction upon encoding on the basis of the prediction mode information PInfo to generate a prediction image. 
     At step S 105 , the arithmetic operation section  114  adds the prediction image obtained by the process at step S 104  to the prediction residual D′ obtained by the process at step S 103  to obtain a decoded image. 
     At step S 106 , the arithmetic operation section  114  outputs the decoded image obtained by the process at step S 105  to the outside of the image decoding apparatus  100 . 
     At step S 107 , the frame memory  115  stores the decoded image obtained by the process at step S 105 . 
     When the process at step S 107  ends, the image decoding process is ended. 
     &lt;Flow of Process of Inverse Transform&gt; 
     Now, an example of a flow of the inverse transform process executed at step S 103  of  FIG. 12  is described with reference to a flow chart of  FIG. 13 . After the inverse transform process is started, at step S 121 , the switch  121  decides whether the transform skip identifier ts_idx is 2D TS (mode of two-dimensional transform skip) or the transform quantization bypass flag transquant_bypass_flag is 1 (true). In the case where it is decided that the transform skip identifier ts_idx is 2D TS or the transform quantization bypass flag is 1 (true), the transform coefficients Coeff_IQ are outputted as a prediction residual D′ to the outside (supplied to the arithmetic operation section  114 ) by the switch  121 , and the inverse transform process ends and the processing returns to  FIG. 12 . 
     On the other hand, in the case where it is decided at step S 121  that the transform skip identifier ts_idx is not 2D TS (mode other than two-dimensional transform skip) and besides the transform quantization bypass flag is 0 (false), the transform coefficients Coeff_IQ are supplied to the inverse secondary transform section  122  by the switch  121 , and the processing advances to step S 122 . 
     At step S 122 , the inverse secondary transform section  122  performs an inverse secondary transform for the transform coefficients Coeff_IQ′ inputted thereto on the basis of the secondary transform identifier st_idx to derive and output the primary transform coefficients Coeff_IS. 
     At step S 123 , the inverse primary transform selection section  131  performs an inverse primary transform selection process to refer to the primary horizontal transform designation flag pt_hor_flag, primary vertical transform designation flag pt_ver_flag, prediction mode information PInfo and transform skip identifier ts_idx to select an inverse primary horizontal transform IPThor and an inverse primary vertical transform IPTver. 
     At step S 124 , the switch  132  decides whether or not the transform skip identifier ts_idx is 1D_V_TS (mode of one-dimensional transform skip in the vertical direction) (ts_idx==1D_V_TS). In the case where the transform skip identifier ts_idx is not 1D_V_TS, the processing advances to step S 125 . 
     At step S 125 , the inverse primary vertical transform section  133  receives, for each transform block, transform coefficients Coeff_IS after the inverse secondary transform and a matrix for an inverse primary vertical transform IPTver as inputs thereto, performs matrix operation, and outputs a result of the matrix operation as transform coefficients Coeff_IPver after the inverse primary vertical transform. After the process at step S 125  ends, the processing advances to step S 126 . 
     On the other hand, in the case where the transform skip identifier ts_idx is 1D_V_TS at step S 124 , the process at step S 125  is omitted and the processing advances to step S 126 . 
     At step S 126 , the switch  134  decides whether or not the transform skip identifier ts_idx is 1D_H_TS (mode of one-dimensional transform skip in the horizontal direction) (ts_idx==1D_H_TS). In the case where it is decided that the transform skip identifier ts_idx is not 1D_H_TS, the processing advances to step S 127 . 
     At step S 127 , the inverse primary horizontal transform section  135  receives, for each transform block, transform coefficients Coeff_IPver after the inverse primary vertical transform and a matrix for an inverse primary horizontal transform IPThor as inputs thereto, performs matrix operation, and outputs a result of the matrix operation as a prediction residual D′. When the process at step S 127  ends, the inverse transform process ends, and the processing returns to  FIG. 12 . 
     On the other hand, in the case where it is decided at step S 126  that the transform skip identifier ts_idx is 1D_H_TS, the process at step S 127  is omitted, and the inverse transform process ends and the processing returns to  FIG. 12 . 
     While the foregoing is a description of the process of the inverse primary transform section  123 , rearrangement of the processing order of the steps or change of the substance of the processes may be performed as far as practicable. For example, the processes at steps S 124  and S 126  may be omitted while it is decided at step S 123  whether or not the transform skip identifier ts_idx is 1D_V_TS (one-dimensional transform skip in the vertical direction). Then, in the case where it is decided at step S 123  that the transform skip identifier ts_idx is 1D_V_TS, a unit matrix may be selected as the inverse primary vertical transform IPTver such that the process at step S 125  is executed. Further, in the case where it is decided at step S 123  that the transform skip identifier ts_idx is 1D_H_TS, a unit matrix may be selected as the inverse primary horizontal transform IPThor such that the process at step S 127  is executed. 
     &lt;Flow of Inverse Primary Transform Selection Process&gt; 
     Now, an example of a flow of the inverse primary transform selection process executed at step S 123  of  FIG. 13  is described with reference to a flow chart of  FIG. 14 . 
     After the inverse primary transform selection process is started, at step S 141 , the inverse primary transform selection section  131  decides whether or not the adaptive primary transform flag apt_flag is 1 (true). In the case where it is decided that the adaptive primary transform flag apt_flag is 1 (true), the processing advances to step S 142 . At step S 142 , the inverse primary transform selection section  131  selects, in regard to each of the inverse primary vertical transform and the inverse primary horizontal transform, a transform set in accordance with the expression (10) given hereinabove on the basis of the prediction mode information PInfo. After the process at step S 142  ends, the processing advances to step S 144 . 
     On the other hand, in the case where it is decided at step S 141  that the adaptive primary transform flag apt_flag is 0 (false), the processing advances to step S 143 . At step S 143 , the inverse primary transform selection section  131  selects a predetermined transform set. After the process at step S 143  ends, the processing advances to step S 144 . 
     At step S 144 , the inverse primary transform selection section  131  refers to the primary horizontal transform set TransformSetH and the primary horizontal transform designation flag pt_hor_flag to select an orthogonal transform to be applied as the inverse primary horizontal transform IPThor in accordance with the expression (11) given hereinabove. 
     At step S 145 , the inverse primary transform selection section  131  refers to the vertical transform set identifier TransformSetV and the primary vertical transform designation flag pt_ver_flag to select an orthogonal transform to be applied as the inverse primary vertical transform IPTver in accordance with the expression (12) given hereinabove. 
     When the process at step S 145  ends, the inverse primary transform selection process ends, and the processing returns to  FIG. 13 . 
     It is to be noted that this inverse primary transform selection process may be subjected to rearrangement of the processing order of the steps or change of the substance of the processes as far as practicable. For example, in the process at step S 144 , in the case of the transform skip identifier ts_idx==TS_1D_H_TS (in the case where a one-dimensional transform in the horizontal direction is to be skipped), an orthogonal transform IDT (unit matrix) that expressly indicates one-dimensional transform skip may be suppressed from being selected. Similarly, in the process at step S 145 , in the case of the transform skip identifier ts_idx==TS_1D_V_TS (in the case where a one-dimensional transform in the vertical direction is to be skipped), orthogonal transform IDT (unit matrix) that expressly indicates one-dimensional transform skip may be suppressed from being selected. 
     By executing the processes in such a manner as described above, the image decoding apparatus  100  can reduce the processing amount of an inverse transform for a residual signal, for which it is desirable to apply transform skip, and suppress reduction of the energy compaction, and can implement enhancement of the encoding efficiency. 
     More particularly, the inverse transform section  113  can decrease the processing amount of an inverse primary transform and suppress reduction of the energy compaction in regard to a residual signal for which it is desirable to skip a one-dimensional transform in the horizontal direction or the vertical direction, and can implement enhancement of the encoding efficiency. Especially, for a residual signal having a characteristic of a step edge by which the continuity of a signal changes rapidly in the horizontal direction, by skipping a one-dimensional transform in the horizontal direction and performing a one-dimensional transform in the vertical direction, non-zero coefficients can be concentrated efficiently on a low frequency region of frequency components in the vertical direction in comparison with those in an alternative case in which a two-dimensional orthogonal transform is performed. In particular, since the energy compaction can be enhanced, enhancement of the encoding efficiency can be implemented. Similarly, for a residual signal having a characteristic of a step edge by which the continuity of a signal changes rapidly in the vertical direction, by skipping a one-dimensional transform in the vertical direction and performing a one-dimensional transform in the horizontal direction, non-zero coefficients can be concentrated efficiently on a low frequency region of frequency components in the horizontal direction in comparison with those in an alternative case in which a two-dimensional orthogonal transform is performed. In particular, since the energy compaction can be enhanced, enhancement of the encoding efficiency can be implemented. 
     2. Second Embodiment 
     &lt;Decoding of Primary Transform Identifier&gt; 
     In JEM3, an adaptive primary transform flag apt_flag is decoded in a CU unit (=PU unit=TU unit), and a primary transform identifier pt_idx and a transform skip flag ts_flag are decoded in each transform block unit included in a CU. Further, from a primary transform identifier pt_idx, a primary horizontal transform designation flag pt_hor_flag and a primary vertical transform designation flag pt_ver_flag are derived in accordance with the expression (19) given below. 
         pt _hor_flag= pt _idx &amp; 0 x 01
 
         pt _ver_flag= pt _idx&gt;&gt;1 (=( pt _idx &amp; 0 x 10)&gt;&gt;1)   (19)
 
     In particular, the primary transform identifier pt_idx has a value of 2 bits, and the upper 1 bit corresponds to the primary vertical transform designation flag pt_ver_flag while the lower 1 bit corresponds to the primary horizontal transform designation flag pt_hor_flag. It is to be noted that, on the encoding side, derivation of the primary transform identifier pt_idx is performed in such a manner as indicated by the expression (20) given below. 
         pt _idx=(( pt _ver_flag)&lt;&lt;1)+ pt _hor_flag   (20)
 
     Accordingly, in the case where changes of (1) to (3) given below are applied to JEM3 of the related art as described hereinabove in connection with the first embodiment, since the transform skip identifier ts_idx decoded in a transform block unit is 1D_H_TS, since the primary horizontal transform designation flag pt_hor_flag is not used, it is redundant to encode/decode this information. Further, in the case where the transform skip identifier ts_idx indicates 1D_V_TS, since the primary vertical transform designation flag pt_ver_flag is not used, it is redundant to encode/decode this information. 
     (1) The transform skip flag ts_flag is expanded to the transform skip identifier ts_idx. 
     (2) When the transform skip identifier ts_idx is 1D_H_TS, an inverse secondary transform and an inverse primary horizontal transform are skipped. 
     (3) When the transform skip identifier ts_idx is 1D_V_TS, an inverse secondary transform and an inverse primary vertical transform are skipped. 
     Accordingly, in order to efficiently encode/decode a primary transform identifier pt_idx, such changes as described below may be applied. 
     (1) In the case of the transform skip identifier ts_idx=NO_TS, on the decoding side, a primary horizontal transform designation flag pt_hor_flag and a primary vertical transform designation flag pt_ver_flag are derived from a primary transform identifier pt_idx in accordance with the expression (19) given hereinabove. On the encoding side, a primary transform identifier pt_idx is derived from a primary horizontal transform designation flag pt_hor_flag and a primary vertical transform designation flag pt_ver_flag in accordance with the expression (20). 
     (2) When the transform skip identifier ts_idx is 1D_H_TS, on the decoding side, the primary vertical transform designation flag pt_ver_flag is derived as pt_ver_flag=pt_idx and the primary horizontal transform designation flag pt_hor_flag is derived as pt_hor_flag=−1 (predetermined value) in accordance with the expression (21) given below. On the encoding side, the primary transform identifier pt_idx is derived as pt_idx=pt_ver_flag in accordance with the expression (22) given below. 
         pt _ver_flag= pt _idx 
         pt _hor_flag=−1 (predetermined value)   (21)
 
         pt _idx=pt_ver_flag   (22)
 
     (3) When the transform skip identifier ts_idx is 1D_V_TS, on the decoding side, the primary horizontal transform designation flag pt_hor_flag is derived as pt_hor_flag=pt_idx and the primary vertical transform designation flag pt_ver_flag is derived as pt_ver_flag=−1 (predetermined value) in accordance with the expression (23) given below. On the encoding side, the primary transform identifier pt_idx is derived as pt_idx=pt_hor_flag in accordance with the expression (24) given below. 
         pt _hor_flag= pt _idx 
         pt _ver_flag=−1 (predetermined value)   (23)
 
         pt _idx=pt_hor_flag   (24)
 
     (4) When the transform skip identifier ts_idx is 2D TS, on the decoding side, the primary horizontal transform designation flag pt_hor_flag is derived as −1 (predetermined value) and the primary vertical transform designation flag pt_ver_flag is derived as pt_ver_flag=−1 (predetermined value) in accordance with the expression (25) given below. On the encoding side, encoding of the primary transform identifier pt_idx is omitted. 
         pt _hor_flag=−1 (predetermined value)
 
         pt _ver_flag=−1 (predetermined value)   (25)
 
     By performing the changes described above, in the case where the transform skip identifier ts_idx is 1D_H_TS or 1D_V_TS, since the primary transform identifier pt_idx is arithmetically decoded/arithmetically encoded as a bin string of 1 bit, bin strings that become a decoding/encoding target can be reduced from those in an alternative case in which the primary transform identifier pt_idx is arithmetically decoded/arithmetically encoded as a bin string of 2 bits. Accordingly, increase of the code amount can be suppressed, and enhancement of the encoding efficiency can be implemented. 
     Further, in regard to a secondary transform, the secondary transform is a transform designed to enhance the energy compaction of transform coefficients to which a two-dimensional orthogonal transform is applied. Accordingly, there is the possibility that application of a secondary transform is applied to transform coefficients to which two-dimensional transform skip or one-dimensional transform skip is applied may conversely decrease the encoding efficiency. Therefore, in the case where the transform skip identifier ts_idx is 2D TS, 1D_H_TS or 1D_V_TS, such a change that an inverse secondary transform is skipped is applied. In this case, it is redundant to encode/decode the secondary transform identifier st_idx that is a control parameter for a (inverse) secondary transform. 
     Therefore, in the case where the transform skip identifier ts_idx is 2D_TS, 1D_H_TS or 1D_V_TS, such a change may be applied that decoding of the secondary transform identifier st_idx is omitted and besides the value of the secondary transform identifier st_idx is estimated to the value (0) that indicates that a secondary transform is not performed. By this change, increase of the processing amount relating to decoding of the secondary transform identifier st_idx can be suppressed. Further, since a secondary transform is not applied to transform coefficients to which two-dimensional transform skip or one-dimensional transform skip is applied, decrease of the encoding efficiency can be prevented. 
     &lt;Decoding of Transform Skip Identifier Ts_Idx&gt; 
       FIG. 15  is a block diagram depicting a principal configuration example relating to decoding of the transform skip identifier ts_idx of the decoding section  111 . As depicted in  FIG. 15 , the decoding section  111  in this case includes a transform skip validity flag decoding section  151 , a maximum transform skip block size decoding section  152 , a transform quantization bypass flag decoding section  153  and a transform skip identifier decoding section  154 . 
     The transform skip validity flag decoding section  151  performs a process relating to decoding of a transform skip validity flag ts_enabled_flag. The maximum transform skip block size decoding section  152  performs a process relating for decoding of a maximum transform skip block size MaxTSSize. The transform quantization bypass flag decoding section  153  performs a process relating to decoding of a transform quantization bypass flag transquant_bypass_flag. The transform skip identifier decoding section  154  performs a process relating to decoding of a transform skip identifier ts_idx. 
     An example of a flow of a decoding process relating to the transform skip identifier ts_idx, which is executed at step S 101  of  FIG. 12  by such a decoding section  111  as described above, is described with reference to a flow chart of  FIG. 16 . 
     After a decoding process is started, at step S 161 , the transform skip validity flag decoding section  151  decodes the transform skip validity flag ts_enabled_flag from a bit string of encoded data #1 and outputs the decoded transform skip validity flag ts_enabled_flag as part of the header information Hinfo. 
     At step S 162 , the maximum transform skip block size decoding section  152  decides whether or not the transform skip validity flag ts_enabled_flag included in the header information HInfo is 1 (true). In the case where it is decided that the transform skip validity flag ts_enabled_flag is 1, the processing advances to step S 163 . 
     At step S 163 , the maximum transform skip block size decoding section  152  decodes a maximum transform skip block size MaxTSSize (or a logarithm log 2MaxTSSize with the base 2) from a bit string of the encoded data #1. After the process at step S 163  ends, the processing advances to step S 164 . On the other hand, in the case where it is decided at step S 162  that the transform skip validity flag ts_enabled_flag is 0, the process at step S 163  is omitted and the processing advances to step S 164 . 
     At step S 164 , the transform quantization bypass flag decoding section  153  decodes the transform quantization bypass flag transquant_bypass_flag from the bit string of the encoded data #1 and outputs the decoded transform quantization bypass flag transquant_bypass_flag as part of the transform information Tinfo. 
     At step S 165 , the transform skip identifier decoding section  154  decides whether or not the transform quantization bypass flag transquant_bypass_flag included in the transform information Tinfo is 1 (true). In the case where it is decided that the transform quantization bypass flag transquant_bypass_flag is 1, the processing advances to S 169 . On the other hand, in the case where it is decided at step S 165  that the transform quantization bypass flag transquant_bypass_flag is 0, the processing advances to step S 166 . 
     At step S 166 , the transform skip identifier decoding section  154  decides whether or not the transform skip validity flag ts_enabled_flag included in the header information Hinfo is 1 (true). In the case where it is decided that the transform skip validity flag ts_enabled_flag is 0, the processing advances to step S 169 . On the other hand, in the case where it is decided at step S 166  that the transform skip validity flag ts_enabled_flag is 1, the processing advances to step S 167 . 
     At step S 167 , the transform skip identifier decoding section  154  decides whether or not a size TBSize of the transform block of a processing target is equal to or smaller than the maximum transform skip block size MaxTSSize (whether or not the logical value of the conditional expression (TBSize&lt;=MaxTSSize) is 1 (true)). It is to be noted that, in the conditional expression (TBSize&lt;=MaxTSSize), TBSize is derived in accordance with the expression (26) given below. In the case of the expression (26), the value of a greater one of the vertical size TBHSize and the horizontal size TBWSize of the transform block is TBSize. 
       TBSize=max(TBWSize,TBHSize)   (26)
 
     In place of the expression (26), the expression (27) given below may be used. In the case of the expression (27), the value obtained by multiplying the vertical size TBHSize and the horizontal size TBWSize of the transform block is TBSize. 
       TBSize=TBWSize*TBHSize   (27)
 
     In the expressions (26) and (27), TBSize, TBWSize and TBHSize may be replaced by logarithms log 2TBSize, log 2TBWSize and log 2TBHSize with the base 2, respectively. In this case, the expression (26) is replaced by the expression (28) given below, and the expression (27) is replaced by the expression (29) given below. It is to be noted that, in the case where logarithms are used, TBSize and the maximum transform skip block size MaxTSSize of the conditional expression (TBSize&lt;=MaxTSSize) given hereinabove are replaced by corresponding logarithms log 2TBSize and log 2MaxTSSize, respectively. 
       log 2TBSize=max(log 2TBWSize,log 2TBHSize)   (28)
 
       log 2TBSize=log 2TBWSize+log 2TBHSize   (29)
 
     In the case where it is decided at step S 167  that the size TBSize of the transform block of the processing target is greater than the maximum transform skip block size MaxTSSize (the logical value of the conditional expression (TBSize&lt;=MaxTSSize) is 0 (false)), the processing advances to step S 169 . On the other hand, in the case where it is decided at step S 167  that the size TBSize of the transform block of the processing target is equal to or smaller than the maximum transform skip block size MaxTSSize (the logical value of the conditional expression (TBSize&lt;=MaxTSSize) is 1 (true)), the processing advances to step S 168 . 
     At step S 168 , the transform skip identifier decoding section  154  decodes the transform skip identifier ts_idx from the bit string of the encoded data #1 and outputs the transform skip identifier ts_idx as part of the transform information Tinfo. When the process at step S 168  ends, the decoding process relating to a transform skip identifier ends and the processing returns to  FIG. 12 . 
     On the other hand, at step S 169 , the transform skip identifier decoding section  154  estimates that the value of ts_idx is NO_TS(=0) omitting decoding of the transform skip identifier ts_idx, and sets the value to ts_idx. In other words, ts_idx becomes ts_idx=NO_TS. When the process at step S 169  ends, the decoding process relating to a transform skip identifier ends and the processing returns to  FIG. 12 . 
     A syntax table in which pseudo codes of the processes at steps S 165  to S 169  are described is depicted in  FIG. 17 . A conditional expression (second row from above) of an if statement to which reference symbol SYN 11  is appended in  FIG. 17  can be represented as in the conditional expression (30) given below, and this particularly is equivalent to a branching decision at steps S 165  to S 167  described hereinabove with reference to  FIG. 16 . Further, the process at step S 168  of  FIG. 16  corresponds to the decoding (encoding) process of syntax ts_flag to which reference symbol SYN 12  is appended in  FIG. 17 . 
       Logical value=( ts _enabled_flag &amp;&amp; !transquant_bypass_flag &amp;&amp; (log 2TBSize&lt;=log 2MaxTSSize))   (30)
 
     While processes relating to decoding of the transform skip identifier ts_idx is described in the foregoing description, rearrangement of the processing order of or change of the substance of the processes at the individual steps may be changed as far as practicable. Further, the conditional expression (30) can be changed in arithmetic operation as far as practicable. 
     By performing the processes in such a manner as described above, in comparison with the case of JEM3, two-dimensional transform skip or one-dimensional transform skip in the horizontal direction or the vertical direction can be adaptively selected in a transform block unit on the decoding side. Accordingly, since a residual signal for which one-dimensional transform skip is more effective than before can be decoded by a mode of one-dimensional transform skip, enhancement of the encoding efficiency can be implemented. 
     &lt;Decoding of Primary Transform Identifier Pt_Idx&gt; 
       FIG. 18  is a block diagram depicting a principal configuration example relating to decoding of a primary transform identifier pt_idx of the decoding section  111 . As depicted in  FIG. 18 , the decoding section  111  includes a primary transform validity flag decoding section  161 , an adaptive primary transform flag decoding section  162  and a primary transform identifier decoding section  163 . 
     The primary transform validity flag decoding section  161  performs a process relating to decoding of a primary transform validity flag pt_enabled_flag. The adaptive primary transform flag decoding section  162  performs a process relating to decoding of a primary transform flag pt_enabled_flag. The primary transform identifier decoding section  163  performs a process relating to decoding of a primary transform identifier pt_idx. 
     An example of a flow of the decoding process relating to a primary transform identifier pt_idx, which is executed at step S 101  of  FIG. 12  by such a decoding section  111  as described above is described with reference to a flow chart of  FIG. 19 . 
     After a decoding process is started, at step S 181 , the primary transform validity flag decoding section  161  decodes the primary transform validity flag pt_enabled_flag from the bit string of the encoded data #1 and outputs the decoded primary transform validity flag pt_enabled_flag as part of the header information Hinfo. 
     At step S 182 , the adaptive primary transform flag decoding section  162  decides whether or not the primary transform validity flag pt_enabled_flag included in the header information Hinfo is 1 (true). In the case where it is decided that the primary transform validity flag pt_enabled_flag is 0 (false), the processing advances to step S 183 . 
     At step S 183 , the adaptive primary transform flag decoding section  162  omits encoding of the adaptive primary transform flag apt_flag and estimates that the value of the flag is 0. After the process at step S 183  ends, the processing advances to step S 193 . 
     On the other hand, in the case where it is decided at step S 182  that the primary transform validity flag pt_enabled_flag is 1 (true), the processing advances to step S 184 . At step S 184 , the adaptive primary transform flag decoding section  162  decodes the adaptive primary transform flag apt_flag from the bit string of the encoded data #1. 
     At step S 185 , the primary transform identifier decoding section  163  decides whether or not the adaptive primary transform flag apt_flag is 1 (true). In the case where it is decided that the adaptive primary transform flag apt_flag is 0 (false), the processing advances to step S 193 . On the other hand, in the case where it is decided at step S 185  that the adaptive primary transform flag apt_flag is 1 (true), the processing advances to step S 186 . 
     At step S 186 , the primary transform identifier decoding section  163  decides whether or not the transform quantization bypass flag transquant_bypass_flag is 1 (true). In the case where it is decided that the transform quantization bypass flag transquant_bypass_flag is 1 (true), the processing advances to step S 193 . On the other hand, in the case where it is decided at step S 186  that the transform quantization bypass flag transquant_bypass_flag is 0 (false), the processing advances to step S 187 . 
     At step S 187 , the primary transform identifier decoding section  163  decides whether or not the transform skip identifier ts_idx is 2D TS (two-dimensional transform skip). In the case where it is decided that the transform skip identifier ts_idx is 2D_TS, the processing advances to step S 193 . On the other hand, in the case where it is decided at step S 187  that the transform skip identifier ts_idx is any other than 2D TS, the processing advances to step S 188 . 
     At step S 188 , the primary transform identifier decoding section  163  decides whether or not both the vertical size (TBHSize) and the horizontal size (TBWSize) of the transform block are equal to or smaller than a maximum adaptive primary transform block size MaxPTSize (max(TBHSize, TBWSize)&lt;=MaxPTSize). In the case where it is decided that at least one of the vertical size or the horizontal size of the transform block is greater than the maximum adaptive primary transform block size MaxPTSize, the processing advances to step S 193 . On the other hand, in the case where it is decided at step S 188  that both the vertical size and the horizontal size of the transform block are equal to or smaller than the maximum adaptive primary transform block size MaxPTSize, the processing advances to step S 189 . 
     At step S 189 , the primary transform identifier decoding section  163  decides whether or not the transform block of the decoding target is a luminance component. In the case where it is decided that the transform block of the decoding target is not a luminance component, the processing advances to step S 193 . On the other hand, in the case where it is decided at step S 189  that the transform block of the decoding target is a luminance component, the processing advances to step S 190 . 
     At step S 190 , the primary transform identifier decoding section  163  refers to the residual information Rinfo to derive a sum total numSig (total number of sig_coeff_flag==1) of non-zero transform coefficients existing in the transform block. 
     At step S 191 , the primary transform identifier decoding section  163  decides whether or not the total number numSig of non-zero transform coefficients is equal to or greater than a predetermined threshold value ptNumSigTH (numSig&gt;=ptNumSigTH). In the case where it is decided that the total number numSig of non-zero transform coefficients is smaller than the predetermined threshold value ptNumSigTH, the processing advances to step S 193 . On the other hand, in the case where it is decided at step S 191  that the total number numSig of non-zero transform coefficients is equal to or greater than the predetermined threshold value ptNumSigTH (numSig&gt;=ptNumSigTH is satisfied), the processing advances to step S 192 . 
     At step S 192 , the primary transform identifier decoding section  163  decodes the primary transform identifier pt_idx from the bit string of the encoded data #1. After the process at step S 192  ends, the processing advances to step S 193 . 
     At step S 193 , the primary transform identifier decoding section  163  refers to the transform skip identifier ts_idx and the primary transform identifier pt_idx decoded already to derive a primary horizontal transform designation flag pt_hor_flag and a primary vertical transform designation flag pt_ver_flag. When the process at step S 193  ends, the decoding process ends and the processing returns to  FIG. 12 . 
     A syntax table in which pseudo codes of the processes at steps S 185  to S 192  described above is described is depicted in  FIG. 17 . A conditional expression of an if statement to which reference symbol SYN 13  is appended in  FIG. 17  (conditional expression (31) given below) is equivalent to a branch decision at steps S 185  to S 192 . Step S 192  corresponds to the decoding process of syntax pt_idx to which reference symbol SYN 14  is appended in  FIG. 17 . It is to be noted that, in the expression (31), the condition at step S 188  is omitted. The expression including the condition at step S 188  is represented by the expression (32). 
       Logical value=(apt_flag !transquant_bypass_flag &amp;&amp;  ts _idx!=2 D _ TS  &amp;&amp; isLuma (cIdx==0) &amp;&amp; numSig&gt;=ptNumSigTH)   (31)
 
       Logical value=(apt_flag !transquant_bypass_flag &amp;&amp;  ts _idx!=2 D _ TS  &amp;&amp; max(TBW,TBH)&lt;=MaxPTSize &amp;&amp; isLuma (cIdx==0) &amp;&amp; numSig&gt;=ptNumSigTH)   (32)
 
     While processes relating to decoding of the primary transform identifier pt_idx are described in the foregoing description, rearrangement of the processing order or the substance of the processes at the individual steps may be changed as far as practicable. 
     Now, a process (primary vertical/horizontal transform designation flag derivation process) relating to derivation of a primary horizontal transform designation flag pt_hor_flag and a primary vertical transform designation flag pt_ver_flag, which is executed at step S 193  of  FIG. 19 , is described with reference to a flow chart of  FIG. 20 . 
     After the primary vertical/horizontal transform designation flag derivation process is started, the primary transform identifier decoding section  163  decides at step S 211  whether or not the transform skip identifier ts_idx is NO_TS (not transform skip). In the case where it is decided that the transform skip identifier ts_idx is NO_TS, the processing advances to step S 212 . 
     At step S 212 , the primary transform identifier decoding section  163  derives the primary horizontal transform designation flag pt_hor_flag and the primary vertical transform flag pt_ver_flag in accordance with the expression (19) given hereinabove. In short, the primary transform identifier pt_idx is processed as a bin string of 2 bits, and a primary horizontal transform designation flag pt_hor_flag and a primary vertical transform flag pt_ver_flag are derived from the 2 bits. When the process at step S 212  ends, the primary vertical/horizontal transform designation flag derivation process ends and the processing returns to  FIG. 19 . 
     In the case where it is decided at step S 212  that the transform skip identifier ts_idx is any other than NO_TS, the processing advances to step S 213 . At step S 213 , the primary transform identifier decoding section  163  decides whether or not the transform skip identifier ts_idx is 2D TS (two-dimensional transform skip). In the case where it is decided that the transform skip identifier ts_idx is 2D TS, the processing advances to step S 214 . 
     At step S 214 , the primary transform identifier decoding section  163  derives the primary horizontal transform designation flag pt_hor_flag and the primary vertical transform flag pt_ver_flag in accordance with the expression (25) given hereinabove. In short, decoding of the primary transform identifier pt_idx is omitted and the primary horizontal transform designation flag pt_hor_flag and the primary vertical transform flag pt_ver_flag are set to predetermined values. When the process at step S 214  ends, the primary vertical/horizontal transform designation flag derivation process ends and the processing returns to  FIG. 19 . 
     In the case where it is decided at step S 213  that the transform skip identifier ts_idx is any other than 2D_TS, the processing advances to step S 215 . At step S 215 , the primary transform identifier decoding section  163  decides whether or not the transform skip identifier ts_idx is 1D_H_TS (one-dimensional transform skip in the horizontal direction). In the case where it is decided that the transform skip identifier ts_idx is 1D_H_TS, the processing advances to step S 216 . 
     At step S 216 , the primary transform identifier decoding section  163  derives the primary horizontal transform designation flag pt_hor_flag and the primary vertical transform flag pt_ver_flag in accordance with the expression (21) given hereinabove. In short, the primary transform identifier pt_idx is processed as a bin string of 1 bit, and the primary horizontal transform designation flag pt_hor_flag is set to the predetermined value and the primary vertical transform flag pt_ver_flag is set to the value of the primary transform identifier pt_idx. When the process at step S 216  ends, the primary vertical/horizontal transform designation flag derivation process ends and the processing returns to  FIG. 19 . 
     In the case where it is decided at step S 215  that the transform skip identifier ts_idx is any other than 1D_H_TS, the processing advances to step S 217 . At step S 217 , the primary transform identifier decoding section  163  derives the primary horizontal transform designation flag pt_hor_flag and the primary vertical transform flag pt_ver_flag in accordance with the expression (23) given hereinabove. In short, the primary transform identifier pt_idx is processed as a bin string of 1 bit, and the primary horizontal transform designation flag pt_hor_flag is set to the value of the primary transform identifier pt_idx and the primary vertical transform flag pt_ver_flag is set to the predetermined value. When the process at step S 217  ends, the primary vertical/horizontal transform designation flag derivation process ends and the processing returns to  FIG. 19 . 
     While the foregoing description is directed to processes relating to derivation of the primary horizontal transform designation flag pt_hor_flag and the primary vertical transform designation flag pt_ver_flag, rearrangement of the processing order of the steps or change of the substance of the processes may be performed as far as practicable. 
     Since, in the case where the transform skip identifier ts_idx is 1D_H_TS or 1D_V_TS, the primary transform identifier pt_idx can be arithmetically decoded as a bin string of 1 bit as described above, the bin string that becomes a decoding target can be reduced from that in an alternative case in which the primary transform identifier pt_idx is arithmetically decoded as a bin string of 2 bits. Accordingly, the processing amount relating to decoding of the primary transform identifier pt_idx can be reduced. Further, since the code amount can be reduced, also enhancement of the encoding efficiency can be implemented. 
     &lt;Decoding of Secondary Transform Identifier st_idx&gt; 
       FIG. 21  is a block diagram depicting a principal configuration example relating to decoding of the secondary transform identifier st_idx of the decoding section  111 . As depicted in  FIG. 21 , the decoding section  111  in this case includes a secondary transform validity flag decoding section  171  and a secondary transform identifier decoding section  172 . 
     The secondary transform validity flag decoding section  171  performs a process relating to decoding of a secondary transform validity flag st_enabled_flag. The secondary transform identifier decoding section  172  performs a process relating to decoding of a secondary transform identifier st_idx. 
     An example of a flow of a decoding process relating to the secondary transform identifier st_idx, which is executed at step S 101  of  FIG. 12  by such a decoding section  111  as described, is described with reference to a flow chart of  FIG. 22 . 
     After the decoding process is started, at step S 231 , the secondary transform validity flag decoding section  171  decodes a secondary transform validity flag st_enabled_flag from the bit string of the encoded data #1 and outputs the secondary transform validity flag st_enabled_flag as part of the header information Hinfo. 
     At step S 232 , the secondary transform identifier decoding section  172  decides whether or not the secondary transform validity flag st_enabled_flag included in the header information Hinfo is 1 (true). In the case where it is decided that the secondary transform validity flag st_enabled_flag is 0 (false), the processing advances to step S 238 . On the other hand, in the case where it is decided at step S 232  that the secondary transform validity flag st_enabled_flag is 1 (true), the processing advances to step S 233 . 
     At step S 233 , the secondary transform identifier decoding section  172  decides whether or not the transform quantization bypass flag transquant_bypass_flag is 1 (true). In the case where it is decided that the transform quantization bypass flag transquant_bypass_flag is 1 (true), the processing advances to step S 238 . On the other hand, in the case where it is decided step S 233  that the transform quantization bypass flag transquant_bypass_flag is 0 (false), the processing advances to step S 234 . 
     At step S 234 , the secondary transform identifier decoding section  172  decides whether or not the transform skip identifier ts_idx is NO_TS (transform skip is not to be performed). In the case where it is decided that the transform skip identifier ts_idx is any other than NO_TS, then processing advances to step S 238 . On the other hand, in the case where it is decided at step S 234  that the transform skip identifier ts_idx is NO_TS, the processing advances to step S 235 . 
     At step S 235 , the secondary transform identifier decoding section  172  refers to the residual information Rinfo to derive a total number numSig (total number of sig_coeff_flag==1) of non-zero transform coefficients existing in the transform block. After the process at step S 235  ends, the processing advances to step S 236 . 
     At step S 236 , the secondary transform identifier decoding section  172  decides whether or not the total number numSig of non-zero transform coefficients is equal to or greater than a predetermine threshold value TH (numSig&gt;=stNumSigTH). In the case where it is decided that the total number numSig of non-zero transform coefficients is smaller than the predetermined threshold value TH (namely, numSig&lt;stNumSigTH is satisfied), the processing advances to step S 238 . 
     On the other hand, in the case where it is decided at step S 236  that the total number numSig of non-zero transform coefficients is later than the predetermined threshold value TH (namely, numSig&gt;=stNumSigTH is satisfied), the processing advances to step S 237 . 
     At step S 237 , the secondary transform identifier decoding section  172  decodes the secondary transform identifier st_idx from the bit string of the encoded data #1. When the process at step S 237  ends, the decoding process ends and the processing returns to  FIG. 12 . 
     On the other hand, at step S 238 , the secondary transform identifier decoding section  172  omits decoding of the secondary transform identifier st_idx and estimates that the value of the secondary transform identifier st_idx is 0, and sets the value to st_idx. In other words, st_idx becomes st_idx=0. When the process at step S 238  ends, the decoding process ends and the processing returns to  FIG. 12 . 
     A syntax table in which pseudo codes at steps S 232  to S 238  (except steps S 235  and S 237 ) are described is depicted in  FIG. 23 . A conditional expression of an if statement to which reference symbol SYN 19  is appended in  FIG. 23  (conditional expression (33) given below) is equivalent to a branch decision at steps S 232  to S 238 . Further, a syntax to which reference symbol SYN 20  is appended in  FIG. 23  corresponds to the process at step S 237  of  FIG. 22 . 
       Logical value=( st _enabled_flag &amp;&amp; (transquant_bypass_flag==0∥ ts _idx== NO _ TS ) &amp;&amp; numSig&gt;=stNumSigTH)   (33)
 
     While processes relating to decoding of the secondary transform identifier st_idx is described in the foregoing description, rearrangement of the processing order or change of the substance of the processes at the individual steps may be changed as far as practicable. Further, the conditional expression (33) can be changed in terms of the arithmetic operation as far as practicable. It is to be noted that, while, in the foregoing description, the control parameter for a secondary transform is the secondary transform identifier st_idx, it may otherwise be a secondary transform flag st_flag. 
     By applying the present technology in such a manner as described above, in comparison with the related art, it is possible to omit, in the case where a secondary transform identifier is to be decoded in a transform block unit, a decoding process of a secondary transform identifier in the case where two-dimensional transform skip or one-dimensional transform skip is to be applied (in the case where ts_idx !=NO_TS). In other words, the processing amount relating to decoding of a secondary transform identifier can be reduced. 
     3. Third Embodiment 
     &lt;Decoding of Primary Horizontal Transform Designation Flag and Primary Vertical Transform Designation Flag&gt; 
     Although it is described in the foregoing description that a primary horizontal transform designation flag pt_hor_flag and a primary vertical transform designation flag pt_ver_flag are derived from a decoded primary transform identifier pt_idx, this is not restrictive. For example, in place of decoding a primary transform identifier pt_idx from encoded data #1, a primary horizontal transform designation flag pt_hor_flag and a primary vertical transform designation flag pt_ver_flag may be decoded directly from encoded data. 
       FIG. 24  is a block diagram depicting a principal configuration example relating to decoding of a primary horizontal transform designation flag pt_hor_flag and a primary vertical transform designation flag pt_ver_flag of the decoding section  111 . As depicted in  FIG. 24 , the decoding section  111  in this case includes a primary transform validity flag decoding section  161 , an adaptive primary transform flag decoding section  162 , a primary horizontal transform designation flag decoding section  181  and a primary vertical transform designation flag decoding section  182 . 
     The primary horizontal transform designation flag decoding section  181  performs a process relating to decoding of the primary horizontal transform designation flag pt_hor_flag. The primary vertical transform designation flag decoding section  182  performs a process relating to decoding of the primary vertical transform designation flag pt_ver_flag. 
     The primary horizontal transform designation flag decoding section  181  decodes a primary horizontal transform designation flag pt_hor_flag from a bit string of encoded data #1 in the case where a condition of an if statement to which reference symbol SYN 15  is appended in the syntax depicted in  FIG. 25  is satisfied, namely, in the case where the transform skip identifier ts_idx is NO_TS or 1D_V_TS (ts_idx==NO_TS II ts_idx==1D_V_TS); the transform quantization bypass flag is 0 (false) (transquant_bypass_flag==0); the transform block is a luminance component (cIdx==0); the adaptive primary transform flag apt_flag is 1 (true) (apt_flag==1); and besides the total number numSig of non-zero transform coefficients in the transform block is equal to or greater than the threshold value ptNumSigTH (numSig&gt;=ptNumSigTH). In any other case, the primary horizontal transform designation flag decoding section  181  sets (estimates) the value of the primary horizontal transform designation flag pt_hor_flag to −1 (predetermined value). 
     Similarly, the primary vertical transform designation flag decoding section  182  decodes a primary vertical transform designation flag pt_ver_flag from a bit string of encoded data #1 in the case where a condition of an if statement to which reference symbol SYN 17  is appended in the syntax depicted in  FIG. 25  is satisfied, namely, in the case where the transform skip identifier ts_idx is NO_TS or 1D_H_TS (ts_idx==NO_TS∥ts_idx==1D_H_TS); the transform quantization bypass flag is 0 (false) (transquant_bypass_flag==0); the transform block is a luminance component (cIdx==0); the adaptive primary transform flag apt_flag is 1 (true) (apt_flag==1); and besides the total number numSig of non-zero transform coefficients in the transform block is equal to or greater than the threshold value ptNumSigTH (numSig&gt;=ptNumSigTH). In any other case, the primary vertical transform designation flag decoding section  182  sets (estimates) the value of the primary vertical transform designation flag pt_ver_flag to −1 (predetermined value). 
     According to the foregoing, in the case where the transform skip identifier ts_idx is 1D_H_TS, decoding of the primary horizontal transform designation flag pt_hor_flag can be omitted. Similarly, in the case where transform skip identifier ts_idx is 1D_V_TS, decoding of the primary vertical transform designation flag pt_ver_flag can be omitted. Accordingly, the processing amount relating to decoding of the primary horizontal transform designation flag pt_hor_flag and the primary vertical transform designation flag pt_ver_flag can be reduced. Further, since increase of the code amount can be suppressed, also enhancement of the encoding efficiency can be implemented. 
     4. Fourth Embodiment 
     &lt;Other Configuration of Inverse Transform Section&gt; 
     Although it is described that, in the first embodiment, in the case where an inverse primary vertical transform is to be skipped, the switch  132  of the inverse primary transform section  123  supplies a primary transform coefficient Coeff_IS to the switch  134 , thereupon, scaling for quantization may be performed. Further, although it is described that, in the first embodiment, in the case where an inverse primary horizontal transform is to be skipped, the switch  134  of the inverse primary transform section  123  outputs transform coefficients Coeff_IPver after the inverse primary vertical transform is outputted as a prediction residual D′ to the outside, thereupon, scaling for quantization may be performed. 
       FIG. 26  is a block diagram depicting a principal configuration example of the inverse transform section  113  in this case. As depicted in  FIG. 26 , the inverse transform section  113  also in this case has a configuration basically similar to that in the case of  FIG. 11 . However, in this case, the inverse primary transform section  123  includes a scaling section  191  and another scaling section  192 . 
     When an inverse primary vertical transform is to be skipped, the switch  132  supplies a primary transform coefficient Coeff_IS to the scaling section  191 . The scaling section  191  performs scaling for the primary transform coefficients Coeff_IS (transform coefficients Coeff_IPver after an inverse primary vertical transform) supplied from the switch  132 . For example, the scaling section  191  performs bit shift arithmetic operation of N (N is a natural number) bits for normalizing the transform coefficients supplied thereto such that the transform coefficients have a norm same as that in the case where the inverse primary vertical transform is carried out. The scaling section  191  supplies the scaled transform coefficients to the switch  134 . 
     In the case where an inverse primary horizontal transform is to be skipped, the switch  134  supplies transform coefficients Coeff_IPver after an inverse primary vertical transform as a prediction residual D′ to the scaling section  192 . Similarly as in the case of the scaling section  191 , the scaling section  192  similarly performs scaling for the transform coefficients Coeff_IPver (prediction residual D′) after the inverse primary vertical transform supplied from the switch  134 . For example, the scaling section  192  performs bit shift arithmetic operation of N (N is a natural number) bits for normalizing the transform coefficients supplied thereto such that they have a norm same as that in an alternative case in which the inverse primary horizontal transform is carried out. The scaling section  192  outputs the scaled prediction residual D′ to the outside. 
     According to the foregoing, since the dynamic range width of transform coefficients can be suppressed within a predetermined range, increase of the load of decoding can be suppressed. 
     5. Fifth Embodiment 
     &lt;Image Encoding Apparatus&gt; 
     Now, encoding for generating encoded data to be decoded in such a manner as described above is described.  FIG. 27  is a block diagram depicting an example of a configuration of an image encoding apparatus that is a form of an image processing apparatus to which the present technology is applied. The image encoding apparatus  300  depicted in  FIG. 27  is an image encoding apparatus corresponding to the image decoding apparatus  100  of  FIG. 10  and generates encoded data (bit stream) to be decoded by the image decoding apparatus  100  by encoding an image by an encoding method corresponding to the decoding method by the image decoding apparatus  100 . For example, the image encoding apparatus  300  incorporates the technology proposed by HEVC or technology proposed by JVET. 
     It is to be noted that principal ones of processing sections, flows of data and so forth are depicted, and not all of such processing sections and so forth are depicted in  FIG. 27 . In other words, in the image encoding apparatus  300 , a processing section that is not depicted as a block in  FIG. 27  may exist, or a process or a flow of data that is not indicated as an arrow mark or the like in  FIG. 27  may exist. 
     As depicted in  FIG. 27 , the image encoding apparatus  300  includes a control section  311 , an arithmetic operation section  312 , a transform section  313 , a quantization section  314 , an encoding section  315 , a dequantization section  316 , an inverse transform section  317 , another arithmetic operation section  318 , a frame memory  319  and a prediction section  320 . 
     The control section  311  segments, on the basis of a block size of a processing unit designated from the outside or designated in advance, a moving image #2 into blocks (CUs, PUs, transform blocks or the like) of a processing unit and inputs an image I corresponding to the segmented blocks to the arithmetic operation section  312 . Further, the control section  311  determines encoding parameters to be supplied to the associated blocks (header information Hinfo, prediction mode information Pinfo, transform information Tinfo and so forth), for example, on the basis of RDO (Rate-Distortion Optimization). The determined encoding parameters are supplied to the associated blocks. In particular, they are supplied in the following manner. 
     The header information Hinfo is supplied to the blocks. The prediction mode information Pinfo is supplied to the encoding section  315  and the prediction section  320 . The transform information Tinfo is supplied to the encoding section  315 , transform section  313 , quantization section  314 , dequantization section  316  and inverse transform section  317 . 
     The arithmetic operation section  312  receives an image I corresponding to a block of a processing unit and a prediction image P supplied from the prediction section  320  as inputs thereto, and subtracts the prediction image P from the image I as indicated by the expression (34) to derive a prediction residual D and supplies the prediction residual D to the transform section  313 . 
         D=I−P    (34)
 
     The transform section  313  performs an inverse process to that of the inverse transform section  317  and receives a prediction residual D and transform information Tinfo as inputs thereto, and applies, on the basis of the transform information Tinfo, a transform to the prediction residual D to derive a transform coefficients Coeff and supplies the transform coefficients Coeff to the quantization section  314 . 
     The quantization section  314  performs an inverse process to that of the dequantization section  316  and receives transform information Tinfo and transform coefficients Coeff as inputs thereto, and scales (quantizes) the transform coefficients Coeff on the basis of the transform information Tinfo and supplies transform coefficients after the quantization, namely, a quantization transform coefficient levels level, to the encoding section  315 . 
     The encoding section  315  performs an inverse process to that of the decoding section  111  ( FIG. 10 ), and converts encoding parameters supplied from the control section  311  (header information, prediction mode information Pinfo and transform information Tinfo) and quantization transform coefficient levels level supplied from the quantization section  314  into syntax values of individual syntax elements in accordance with a definition of a syntax table and variable length encodes (for example, arithmetically encodes) the syntax values to generate a bit string. 
     It is to be noted that the encoding section  315  derives residual information RInfo from quantization transform coefficient levels level and variable length encodes the residual information RInfo to generate a bit string. Further, the encoding section  315  multiplexes the bit strings of the variable length encoded syntax elements to generate encoded data #1 and outputs the encoded data #1. 
     The dequantization section  316  is a processing section similar to the dequantization section  112  ( FIG. 10 ) and performs a process similar to that of the dequantization section  112  in the image encoding apparatus  300 . The inverse transform section  317  is a processing section similar to the inverse transform section  113  ( FIG. 10 ) and performs a process similar to that of the inverse transform section  113  in the image encoding apparatus  300 . The frame memory  319  is a processing section similar to the frame memory  115  ( FIG. 10 ) and performs a process similar to that of the frame memory  115  in the image encoding apparatus  300 . The prediction section  320  is a processing section similar to the prediction section  116  ( FIG. 10 ) and performs a similar process to that of the prediction section  116  in the image encoding apparatus  300 . 
     &lt;Transform Section&gt; 
       FIG. 28  is a block diagram depicting a principal configuration example of the transform section  313 . As depicted in  FIG. 28 , the transform section  313  includes a switch  331 , a primary transform section  332  and a secondary transform section  333 . 
     The switch  331  receives a prediction residual D and a transform skip identifier ts_idx as inputs thereto and supplies the prediction residual D to the primary transform section  332  in the case where the value of the transform skip identifier ts_idx is NO_TS(=0), 1D_H_TS (=2) or 1D_V_TS(=3) (in the case where transform skip is not applied or one-dimensional transform skip is applied to one of the horizontal and vertical directions). On the other hand, in the case where the value of the transform skip identifier ts_idx is 2D TS(=1) (in the case where it is indicated that two-dimensional transform skip is applied), the switch  331  skips the primary transform section  332  and the secondary transform section  333  and outputs the prediction residual D as transform coefficients Coeff to the outside (quantization section  314 ) of the transform section  313 . 
     For example, in the case where the transform skip flag ts_idx is 2D TS(=1) and the prediction residual D inputted to the transform section  313  is the prediction residual D of a 4×4 matrix=[[255, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]], the switch  331  outputs the prediction residual D as transform coefficients Coeff_IQ. In particular, the transform coefficients Coeff become Coeff=[[255, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]. Accordingly, a primary transform and a secondary transform can be skipped. Especially, it is possible to perform, for a sparse residual signal to which it is desirable to apply two-dimensional transform skip and in which the number of non-zero coefficients is small, a transform process that achieves reduction of the processing amount of a transform and suppresses the energy compaction, and a transform process whose encoding efficiency is enhanced can be performed. 
     The primary transform section  332  receives a primary horizontal transform designation flag pt_hor_flag, a primary vertical transform designation flag pt_ver_flag, prediction mode information PInfo, a transform skip identifier ts_idx and a prediction residual D as inputs thereto. The primary transform section  332  selects the prediction mode information PInfo, the transform skip identifier, a matrix PThor for a primary horizontal transform designated by the primary horizontal transform designation flag pt_hor_flag and a matrix PTver for a primary vertical transform designated by the primary vertical transform designation flag pt_ver_flag, performs a primary horizontal transform and a primary vertical transform with the selected matrices for the primary transforms in the directions for the prediction residual D to derive a prediction residual D′ after the primary transforms, and outputs the prediction residual D′. 
     The secondary transform section  333  receives a secondary transform identifier st_idx, a scan identifier scanIdx indicative of a scanning method of a transform coefficient and transform coefficients Coeff_P after a primary transform as inputs thereto, derives transform coefficients Co eff (also referred to as secondary transform coefficients Coeff S) after a secondary transform, and outputs the transform coefficients Co eff. 
     More particularly, in the case where the secondary transform identifier st_idx indicates that a secondary transform is to be applied (st_idx&gt;0), the secondary transform section  333  executes a process of a secondary transform corresponding to the secondary transform identifier st_idx for the primary transform coefficients Coeff_P and outputs transform coefficients Coeff S after the secondary transform. On the other hand, in the case where the secondary transform identifier st_idx indicates that a secondary transform is not to be applied (st_idx==0), the secondary transform section  333  skips a secondary transform and outputs the transform coefficients Coeff_P after the primary transform as secondary transform coefficients Coeff_S after the secondary transform. 
     As depicted in  FIG. 28 , the primary transform section  332  includes a primary transform selection section  341 , a switch  342 , a primary horizontal transform section  343 , another switch  344  and a primary vertical transform section  345 . 
     The primary transform selection section  341  receives a primary horizontal transform designation flag pt_hor_flag, a primary vertical transform designation flag pt_ver_flag, prediction mode information PInfo and a transform skip identifier ts_idx as inputs thereto, reads out the prediction mode information PInfo, the transform skip identifier ts_idx, a matrix PThor for a primary horizontal transform designated by the primary horizontal transform designation flag pt_hor_flag and a matrix PTver for a primary vertical transform designated by the primary vertical transform designation flag pt_ver_flag from an internal memory (not depicted) of the primary transform selection section  341 , and outputs them to the primary horizontal transform section  343  and the primary vertical transform section  345 , respectively. 
     The primary transform selection section  341  selects a transform set TransformSet including orthogonal transforms that become candidates for a primary transform for each of the horizontal direction and the vertical direction similarly as in the case of the inverse primary transform selection section  131 . It is to be noted that this process is same as the process of the inverse primary transform selection section  131 , and therefore, description of this is omitted. 
     Similarly as in the selection process of an inverse primary horizontal transform IPThor and an inverse primary vertical transform IPTver by the inverse primary transform selection section  131 , the primary transform selection section  341  selects an orthogonal transform to be used for a primary transform in the horizontal direction from within the selected transform set TransformSet using the primary horizontal transform designation flag pt_hor_flag, and selects an orthogonal transform to be used for a primary transform in the vertical direction using the primary vertical transform designation flag pt_ver_flag. As a particular process, for the selection process of an inverse primary horizontal transform IPThor and a primary vertical transform IPTver by the inverse primary transform selection section  131 , the following replacement is performed for interpretation:
         inverse primary horizontal transform IPThor→primary horizontal transform PThor   inverse primary vertical transform IPTver→primary vertical transform PTver   inverse primary transform→primary transform       

     The switch  342  receives a prediction residual D and a transform skip identifier ts_idx as inputs thereto. In the case where the value of the transform skip identifier ts_idx is 1D_H_TS(=2), namely, indicates that a one-dimensional transform in the horizontal direction is to be skipped (ts_idx==1D_H_TS) (to skip a primary horizontal transform), the switch  342  skips the primary horizontal transform section  343  and outputs the prediction residual D as transform coefficients Coeff_Phor after a primary horizontal transform to the switch  344 . On the other hand, in the case where the value of the transform skip identifier ts_idx is any other than 1D_H_TS(=2) (ts_idx !=1D_H_TS) (indicates that a primary horizontal transform is not to be skipped), the switch  342  outputs the prediction residual D to the primary horizontal transform section  343 . 
     The primary horizontal transform section  343  receives, for each transform block, a prediction residual D and a matrix for a primary horizontal transform PThor as inputs thereto, and performs matrix operation as indicated by the expression (35) given below and outputs a result of the matrix operation as transform coefficients Coeff_Phor after the primary horizontal transform. It is to be noted that the primary horizontal transform PThor is a matrix having a transformation basis as a column vector. 
       Coeff_Phor=( D−PT hor)&gt;&gt; s 3   (35)
 
     Here, the operator “·” represents a matrix product (inner product), and the operator “&gt;&gt;” represents an operation for performing right shift operation for each element. According to the expression (35), each value of the transform coefficients Coeff_Phor is obtained by performing, for each element of the matrix product of the prediction residual D and the primary horizontal transform PThor, right shift operation with a predetermined scaling parameter s3. It is to be noted that the scaling parameter s3 is used to normalize a matrix operation result of D*PThor so as to fit in the bit depth of an intermediate buffer. The value of the scaling parameter s3 is determined, for example, from the bit depth BitDepthbuff of the intermediate buffer and the worst case MaxBitDepth(D*PThor) of the bit depth of the matrix operation of D*PThor as indicated by the following expression (36) 
         s 3=max(0,MaxBitDepth( D·PT hor)−BitDpethbuff)   (36)
 
     For example, in the case where the value range of D·PThor is −2**22 to 2**22−1, namely, in the case where the bit depth of MaxBitDepth(D*PThor) is 23 bits and the value range of a value that can be stored into the intermediate buffer is −2**15 to 2**15−1, namely, in the case where the bit depth of the intermediate buffer is 16 bits, the scaling parameter s3 is s3=7 bits (=max(0, 23-16)=max(0, 7)). On the other hand, in the case where the bit depth of MaxBitDepth(D*PThor) is 23 bits and the value range of a value that can be stored into the intermediate buffer is −2**31 to 2**31−1, namely, in the case where the bit depth of the intermediate buffer is 32 bits, the scaling parameter s3 becomes s3=0 bit (=max(0, 23-32)=max(0, − 9 )). That the scaling parameter s3 is 0 represents that, since the value of the bit depth of the intermediate buffer is sufficiently high, even if element values of the matrix product of D*PThor are not normalized, they can be stored into the intermediate buffer without overflowing. 
     It is to be noted that the scaling parameter s3 may be a fixed value determined in advance assuming that the value range of D*PThor is known. Further, in order to reduce a clip error by right shift operation of the expression (36) described hereinabove, after the matrix product, a predetermined offset value o3 may be added for each element. At this time, the offset value o3 is represented by the following expression (37) using the scaling parameter s3. 
         o 3=( s 3&gt;0?1&lt;&lt;( s 3−1):0)   (37)
 
     The switch  344  receives a transform coefficients Coeff_Phor after a primary horizontal transform and a transform skip identifier ts_idx as inputs thereto. In the case where the value of the transform skip identifier ts_idx is 1D_V_TS(=3), namely, indicates that a one-dimensional transform in the vertical direction is to be skipped (ts_idx==1D_V_TS) (to skip a primary vertical transform), the switch  344  skips the primary vertical transform section  345  and outputs the transform coefficients Coeff_Phor after the primary horizontal transform as transform coefficients Coeff_P after a primary transform to the outside (secondary transform section  333 ). On the other hand, in the case where the value of the transform skip identifier ts_idx is any other than 1D_V_TS(=2) (ts_idx !=1D_V_TS) (in the case where it is indicated that a primary vertical transform is not to be skipped), the switch  344  outputs the transform coefficients Coeff_Phor after the primary horizontal transform to the primary vertical transform section  345 . 
     The primary vertical transform section  345  receives, for each transform block, transform coefficients Coeff_Phor after a primary horizontal transform and a matrix for a primary vertical transform PTver as inputs thereto, performs matrix operation as indicated by the expression (38) given below, and outputs a result of the matrix operation as transform coefficients Coeff_P after a primary transform. It is to be noted that the primary vertical transform PTver is a matrix having a transform basis as a row vector. 
       Coeff_ P =( PT ver*Coeff_ P hor)&gt;&gt; s 3   (38)
 
     Here, the operator “·” represents a matrix product (inner product), and the operator “&gt;&gt;” represents an operation for performing right shift operation for each element. According to the expression (38), each value of the transform coefficients Coeff_P is obtained by performing, for each element of the matrix product of the transform coefficients Coeff_Phor after a primary horizontal transform and the matrix PTver of the primary vertical transform, right shift arithmetic operation with a predetermined scaling parameter s3. It is to be noted that the scaling parameter s4 is used to normalize matrix operation results of PTver·Coeff_Phor so as to fit in a desire bit depth. The value of the scaling parameter s4 is determined from a desired bit depth BitDepthout and the worst case MaxBitDepth(PTver·Coeff_Phor) of the bit depth of the matrix operation of PTver·Coeff_Phor in accordance with the following expression (39). 
         s 4=max(0,MaxBitDepth( PT ver*Coeff_ P hor)−BitDpethout)   (39)
 
     For example, in the case where the value range of PTver*Coeff_Phor is −2**22 to 2**22−1, namely, in the case where the bit depth of MaxBitDepth(PTver*Coeff_Phor) is 23 bits and the value range of the value that can be taken with a desired bit depth is −2**15 to 2**15−1, namely, in the case where the desired bit depth is 16 bits, the scaling parameter s4 becomes s4=7 bits (=max(0, 23-16)=max(0, 7)). Further, in the case where the bit depth of MaxBitDepth(PTver*Coeff_Phor) is 23 bits and the value range of the value that can be taken with a desired bit depth is −2**31 to 2**31−1, namely, in the case where the desired bit depth is 32 bits, the scaling parameter s4 becomes s4=0 bit (=max(0, 23-32)=max(0, −9)). Since that the scaling parameter s4 is 0 represents that, since the value of the desired bit depth is sufficiently high, even if the pixel values of the matrix product of PTver*Coeff_Phor are not normalized, the pixel values fit in the desired bit depth. 
     It is to be noted that, in order to reduce a clip error by right shift operation of the expression (38) described hereinabove, after the matrix product, a predetermined offset value o4 may be added for each element. At this time, the offset value o3 is represented by the following expression (40) using the scaling parameter s4. 
         o 4=( s 4&gt;0?1&lt;&lt;( s 4−1):0)   (40)
 
     As described above, the primary transform section  332  can perform, for a residual signal in regard to which it is desirable to skip a one-dimensional transform in the horizontal direction or the vertical direction, a primary transform process that decreases the processing amount of a primary transform and prevents decrease of energy compaction to improve the encoding efficiency. Especially for a residual signal having a characteristic of a step edge in which the continuity of a signal changes rapidly in the horizontal direction, by skipping a one-dimensional transform in the horizontal direction and performing a one-dimensional transform in the vertical direction, non-zero coefficients can be concentrated efficiently on a low frequency region of frequency components in the vertical direction in comparison with those in the case where a two-dimensional orthogonal transform is performed. In particular, since energy compaction can be increased, the enhancement of the encoding efficiency can be implemented. Similarly, for a residual signal having a characteristic of a step edge in which the continuity of a signal changes rapidly in the vertical direction, by skipping a one-dimensional transform in the vertical direction and performing a one-dimensional transform in the horizontal direction, non-zero coefficients can be concentrated efficiency on a low frequency region of frequency components in the horizontal direction in comparison with those in an alternative case in which a two-dimensional orthogonal transform is performed. In particular, since energy compaction can be increased, enhancement of the encoding efficiency can be implemented. 
     &lt;Flow of Image Encoding Process&gt; 
     Now, a flow of processes executed by such an image encoding apparatus  300  as described above is described. First, an example of a flow of an image encoding process is described with reference to a flow chart of  FIG. 29 . 
     After the image encoding process is started, at step S 301 , the control section  311  performs an encoding controlling process and performs block segmentation, setting of encoding parameters and so forth. 
     At step S 302 , the prediction section  320  performs a prediction process to generate a prediction image of an optimum prediction mode and so forth. For example, in this prediction process, the prediction section  320  performs intra prediction to generate a prediction image of an optimum intra prediction mode and so forth and performs inter prediction to generate a prediction image of an optimum inter prediction mode and so forth, and selects an optimum prediction mode from among them on the basis of a cost function value and so forth. 
     At step S 303 , the arithmetic operation section  312  arithmetically operates the difference between the input image and the prediction image of the optimum mode selected by the prediction process at step S 302 . In short, the arithmetic operation section  312  generates a prediction residual D between the input image and the prediction image. The prediction residual D calculated in this manner is reduced in data amount in comparison with the original image data. Accordingly, the data amount can be compressed in comparison with that in an alternative case in which the image is encoded as it is. 
     At step S 304 , the transform section  313  performs a transform process for the prediction residual D generated by the process at step S 303  to derive transform coefficients Coeff. It is to be noted that this transform process is an inverse process to the inverse transform process at step S 307  and is an inverse process to the inverse transform process executed in the image decoding process described hereinabove. Details of the process at step S 304  are hereinafter described. 
     At step S 305 , the quantization section  314  quantizes, by using quantization parameters calculated by the control section  311  or the like, the transform coefficients Coeff obtained by the process at step S 304  to derive quantization transform coefficient levels level. 
     At step S 306 , the dequantization section  316  dequantizes the quantization transform coefficient levels level generated by the process at step S 305  in accordance with a characteristic corresponding to the characteristic of the quantization a step S 305  to derive transform coefficients Coeff_IQ. 
     At step S 307 , the inverse transform section  317  inversely transform the transform coefficients Coeff_IQ obtained by the process at step S 306  by a method corresponding to the transform process at step S 304  to derive a prediction residual D′. It is to be noted that this inverse transform process is executed similarly to the inverse transform process executed in the image decoding process described hereinabove. 
     At step S 308 , the arithmetic operation section  318  adds the prediction image obtained by the prediction process at step S 302  to the prediction residual D′ derived by the process at step S 307  to generate a decoded image that is decoded locally. 
     At step S 309 , the frame memory  319  stores the decoded image decoded locally obtained by the process at step S 308 . 
     At step S 310 , the encoding section  315  encodes the quantization transform coefficient levels level obtained by the process at step S 305 . For example, the encoding section  315  encodes the quantization transform coefficient levels level, which is information relating to the image, by arithmetic encoding or the like to generate encoded data. Further, at this time, the encoding section  315  encodes various encoding parameters (header information Hinfo, prediction mode information Pinfo and transform information Tinfo). Furthermore, the encoding section  315  derives residual information RInfo from the quantization transform coefficient levels level and encodes the residual information RInfo. 
     At step S 311 , the encoding section  315  summarizes the encoded data of the various information generated in this manner and outputs them as a bit stream to the outside of the image encoding apparatus  300 . This bit stream is transmitted to the decoding side, for example, through a transmission line or a recording medium. When the process at step S 311  ends, the image encoding process ends. 
     It is to be noted that the processing unit in each of the processes is arbitrary and the processing units may not be same with each other. Accordingly, the processes at the steps can be suitably executed also in parallel with a process at some other step or the like or with the processing order rearranged. 
     &lt;Flow of Transform Process&gt; 
     Now, an example of a flow of the transform process executed at step S 304  of  FIG. 29  is described with reference to a flow chart of  FIG. 30 . After the transform process is started, the switch  331  decides at step S 331  whether the transform skip identifier ts_idx is 2D TS (mode of two-dimensional transform skip) or the transform quantization bypass flag transquant_bypass_flag is 1 (true). In the case where it is decided that the transform skip identifier ts_idx is 2D TS or the transform quantization bypass flag is 1 (true), the switch  331  outputs the prediction residual D as transform coefficients Coeff to the outside. In particular, the transform process ends, and the processing returns to  FIG. 29 . 
     On the other hand, in the case where it is decided at step S 331  that the transform skip identifier ts_idx is not 2D TS (mode other than two-dimensional transform skip) and besides the transform quantization bypass flag is 0 (false), the switch  331  outputs the prediction residual D to the primary transform section  332 . In short, the processing advances to step S 332 . 
     At step S 332 , the primary transform selection section  341  refers to the primary horizontal transform designation flag pt_hor_flag, primary vertical transform designation flag pt_ver_flag, prediction mode information PInfo and transform skip identifier ts_idx to select a primary horizontal transform PThor and a primary vertical transform PTver and supplies them to the primary horizontal transform section  343  and the primary vertical transform section  345 , respectively. 
     At step S 333 , the switch  342  decides whether or not the transform skip identifier ts_idx is 1D_H_TS (mode of one-dimensional transform skip in the horizontal direction) (ts_idx==1D_H_TS). In the case where it is decided that the transform skip identifier ts_idx is not 1D_H_TS, the processing advances to step S 334 . At step S 334 , the primary horizontal transform section  343  receives, for each transform block, the prediction residual D and a matrix for the primary horizontal transform PThor as inputs thereto, perform matrix operation, and outputs a result of the matrix operation as transform coefficients Coeff_Phor after the primary horizontal transform. After the process at step  334  ends, the processing advances to step S 335 . 
     On the other hand, in the case where it is decided at step S 333  that the transform skip identifier ts_idx is 1D_H_TS, the switch  342  skips the primary horizontal transform section  343  and supplies the prediction residual D as transform coefficients Coeff_Phor after the primary horizontal transform to the switch  344 . In short, the process at step S 334  is omitted, and the processing advances to step S 335 . 
     At step S 335 , the switch  344  decides whether or not the transform skip identifier ts_idx is 1D_V_TS (mode of one-dimensional transform skip in the vertical direction) (ts_idx==1D_V_TS). In the case where it is decided that the transform skip identifier ts_idx is not 1D_V_TS, the processing advances to step S 336 . At step S 336 , the primary vertical transform section  345  performs a primary vertical transform. In particular, the primary vertical transform section  345  receives, for each transform block, the transform coefficients Coeff_Phor after the primary horizontal transform and a matrix for the primary vertical transform PTver as inputs thereto, perform matrix operation, and outputs a result of the matrix operation as transform coefficients Coeff_P after the primary horizontal transform. After the process at step  336  ends, the processing advances to step S 337 . 
     On the other hand, in the case where it is decided at step S 335  that the transform skip identifier ts_idx is 1D_V_TS, the switch  344  skips inputting to the primary vertical transform section  345  and supplies the transform coefficients Coeff_Phor after the primary horizontal transform as transform coefficients Coeff_P after the primary transform. In short, the process at step S 336  is omitted, and the processing advances to step S 337 . 
     At step S 337 , the secondary transform section  333  performs a secondary transform for the transform coefficients Coeff_P inputted thereto on the basis of the secondary transform identifier st_idx to derive transform coefficients Coeff and outputs the transform coefficients Coeff. When the process at step S 337  ends, the transform process ends and the processing returns to  FIG. 29 . 
     &lt;Flow of Primary Transform Selection Process&gt; 
     Now, an example of a flow of the primary transform selection process executed at step S 332  of  FIG. 30  is described with reference to a flow chart of  FIG. 31 . 
     After the primary transform selection process is started, at step S 351 , the primary transform selection section  341  decides whether or not the adaptive primary transform flag apt_flag is 1 (true). In the case where it is decided that the adaptive primary transform flag apt_flag is 1 (true), the processing advances to step S 352 . At step S 352 , the primary transform selection section  341  selects a transform set for each of a primary vertical transform and a primary horizontal transform on the basis of the prediction mode information PInfo. After the process at step S 352  ends, the processing advances to step S 354 . 
     On the other hand, in the case where it is decided at step S 351  that the adaptive primary transform flag apt_flag is 0 (false), the processing advances to step S 353 . At step S 353 , the primary transform selection section  341  selects a predetermined transform set. After the process at step S 353  ends, the processing advances to step S 354 . 
     At step S 354 , the primary transform selection section  341  refers to the horizontal transform set identifier TransformSetH and the primary horizontal transform designation flag pt_hor_flag to select an orthogonal transform to be applied as the primary horizontal transform PThor. 
     At step S 355 , the primary transform selection section  341  refers to the vertical transform set identifier TransformSetV and the primary vertical transform designation flag pt_ver_flag to select an orthogonal transform to be applied as the primary vertical transform PTver. 
     When the process at step S 355  ends, the primary transform selection process ends and the processing returns to  FIG. 30 . 
     It is to be noted that also the processes described above may be subjected to rearrangement of the processing order of the steps or change of the substance of the processes as far as practicable. For example, the processes at steps S 333  and S 335  may be omitted while, in the case where the transform skip identifier ts_idx is 1D_H_TS (one-dimensional transform skip in the horizontal direction) at step S 331 , a unit matrix is selected as the primary horizontal transform PThor and the process at step S 334  is executed. On the other hand, in the case where the transform skip identifier ts_idx is 1D_V_TS (one-dimensional transform skip in the vertical direction), a unit matrix may be selected as the primary vertical transform PTver, whereafter the process at step S 336  is executed. 
     According to the foregoing, the transform section  313  provided in the image encoding apparatus  300  can reduce the processing amount for a transform and suppress decrease of the energy compaction in regard to a residual signal, for which it is desirable to apply transform skip, and suppress reduction of the energy compaction, and can perform an inverse transform process that improved in encoding efficiency. More particularly, the transform section  313  can reduce the processing amount for a primary transform and suppress decrease of the energy compaction in regard to a residual signal, for which it is desirable to skip one-dimensional transform in the horizontal direction or the vertical direction, and can perform an inverse primary transform process that is improved in encoding efficiency. 
     Especially, in regard to a residual signal having a characteristic of a step edge in which the continuity of a signal varies rapidly in the horizontal direction, by skipping a one-dimensional transform in the horizontal direction and performing a one-dimensional transform in the vertical direction, non-zero coefficients can be concentrated efficiently on a low frequency region of frequency components in the vertical direction in comparison with those in the case where a two-dimensional orthogonal transform is performed. In particular, since energy compaction can be increased, the encoding efficiency can be enhanced. 
     Similarly, for a residual signal having a characteristic of a step edge in which the continuity of a signal changes rapidly in the vertical direction, by skipping a one-dimensional transform in the vertical direction and performing a one-dimensional transform in the horizontal direction, non-zero coefficients can be concentrated efficiently on a low frequency region of frequency components in the horizontal direction in comparison with those in an alternative case in which a two-dimensional orthogonal transform is performed. In particular, energy compaction can be increased, and the encoding efficiency can be enhanced. 
     6. Sixth Embodiment 
     &lt;Encoding of Primary Transform Identifier&gt; 
     The transform skip identifier ts_idx and the primary transform identifier pt_idx in the image decoding apparatus  100  described above correspond to an inverse process to the decoding process described hereinabove. Accordingly, also the image encoding process is subjected to such control as described below on the basis of the parameters similarly as in the case of the image decoding process. 
     (1) The transform skip flag ts_flag is expanded to the transform skip identifier ts_idx. 
     (2) When the transform skip identifier ts_idx is 1D_H_TS, a primary horizontal transform and a secondary transform are skipped. 
     (3) When the transform skip identifier ts_idx is 1D_V_TS, a primary vertical transform and a secondary transform are skipped. 
     In this case, in the case where the transform skip identifier ts_idx encoded in a transform block unit is 1D_H_TS, since the primary horizontal transform designation flag pt_hor_flag is not used, it is redundant to encode this information. On the other hand, in the case where the transform skip identifier ts_idx is 1D_V_TS, since the primary vertical transform designation flag pt_ver_flag is not used, it is redundant to encode this information. Accordingly, the following changes are applied in order to efficiently encode the primary transform identifier pt_idx. 
     (1) In the case where the transform skip identifier ts_idx=NO_TS, on the encoding side, a primary transform identifier pt_idx is derived from a primary horizontal transform designation flag pt_hor_flag and a primary vertical transform designation flag pt_ver_flag in accordance with the expression (20) given hereinabove. 
     (2) In the case where the transform skip identifier ts_idx is 1D_H_TS, on the encoding side, the primary transform identifier pt_idx is derived as pt_idx=pt_ver_flag in accordance with the expression (22) given hereinabove. 
     (3) In the case where the transform skip identifier ts_idx is 1D_V_TS, on the encoding side, the primary transform identifier pt_idx is derived as pt_idx=pt_hor_flag in accordance with the expression (24) given hereinabove. 
     (4) In the case where the transform skip identifier ts_idx is 2D_TS, on the encoding side, encoding of the primary transform identifier pt_idx is omitted. 
     By such changes as described above, in the case where the transform skip identifier ts_idx is 1D_H_TS or 1D_V_TS, since the primary transform identifier pt_idx is arithmetically transformed as a bin string of 1 bit, the bin string that becomes an encoding target can be reduced in comparison with that in an alternative case in which the primary transform identifier pt_idx is arithmetically encoded as a bin string of 2 bits. Accordingly, increase of the code amount can be suppressed and the encoding efficiency can be increased. 
     Further, regarding a secondary transform, since the transform skip identifier ts_idx is 2D TS, 1D_H_TS or 1D_V_TS, since a secondary transform is skipped, it is redundant to encode the secondary transform identifier st_idx that is a control parameter for the secondary transform. Therefore, in the case where the transform skip identifier ts_idx is 2D TS, 1D_H_TS or 1D_V_TS, by applying such a change that encoding of the transform skip identifier ts_idx is omitted, increase of the processing amount relating to encoding of the secondary transform identifier st_idx can be suppressed. 
     &lt;Encoding of Transform Skip Identifier ts_idx&gt; 
       FIG. 32  is a block diagram depicting a principal configuration example relating to encoding of the transform skip identifier ts_idx of the encoding section  315 . As depicted in  FIG. 32 , the encoding section  315  includes a transform skip validity flag encoding section  361 , a maximum transform skip block size encoding section  362 , a transform quantization bypass flag encoding section  363  and a transform skip identifier encoding section  364 . 
     The transform skip validity flag encoding section  361  performs a process relating to encoding of the transform skip validity flag ts_enabled_flag. The maximum transform skip block size encoding section  362  performs a process relating to encoding of the maximum transform skip block size MaxTSSize. The transform quantization bypass flag encoding section  363  performs a process relating to encoding of the transform quantization bypass flag transquant_bypass_flag. The transform skip identifier encoding section  364  performs a process relating to encoding of the transform skip identifier ts_idx. 
     An example of a flow of the encoding process relating to the transform skip identifier ts_idx, which is executed at step S 310  of  FIG. 29  by the encoding section  315 , is described with reference to a flow chart of  FIG. 33 . 
     After the encoding process is started, at step S 371 , the transform skip validity flag encoding section  361  variable length encodes the transform skip validity flag ts_enabled_flag included in the header information HInfo to generate a bit string and outputs the bit string. 
     At step S 372 , the maximum transform skip block size encoding section  362  decides whether or not the transform skip validity flag ts_enabled_flag is 1 (true). In the case where it is decided that the transform skip validity flag ts_enabled_flag is 1 (true), the processing advances to step S 373 . 
     At step S 373 , the maximum transform skip block size encoding section  362  variable length encodes a maximum transform skip block size MaxTSSize included in the header information Hinfo (or a logarithm log 2MaxTSSize with the base 2) to generate a bit string and outputs the bit string. After the process at step S 373  ends, the processing advances to step S 374 . On the other hand, in the case where it is decided at step S 372  that the transform skip validity flag ts_enabled_flag is 0 (false), the process at step S 373  is omitted and the processing advances to step S 374 . 
     At step S 374 , the transform quantization bypass flag encoding section  363  variable length encodes a transform quantization bypass flag transquant_bypass_flag included in the header information HInfo to generate a bit string and outputs the bit string. 
     At step S 375 , the transform skip identifier encoding section  364  decides whether or not the transform quantization bypass flag transquant_bypass_flag included in the transform information Tinfo is 1 (true). In the case where it is decided that the transform quantization bypass flag transquant_bypass_flag is 1 (true), encoding the transform skip identifier ts_idx is omitted and the encoding process is ended, and the processing returns to  FIG. 29 . On the other hand, in the case where it is decided at step S 375  that the transform quantization bypass flag transquant_bypass_flag is 0 (false), the processing advances to step S 376 . 
     At step S 376 , the transform skip identifier encoding section  364  decides whether or not the transform skip validity flag ts_enabled_flag included in the header information HInfo is 1 (true). In the case where it is decided that the transform skip validity flag ts_enabled_flag is 0 (false), encoding of the transform skip identifier ts_idx is omitted and the encoding process is ended, and the processing returns to  FIG. 29 . On the other hand, in the case where it is decided that the transform skip validity flag ts_enabled_flag is 1 (true), the processing advances to step S 377 . 
     At step S 377 , the transform skip identifier encoding section  364  decides whether or not the size TBSize of the transform block of the processing target is equal to or smaller than the maximum transform skip block size MaxTSSize, or in other words, whether or not the logical value of the conditional expression (TBSize&lt;=MaxTSSize) is 1 (true). 
     It is to be noted that, in the conditional expression (TBSize&lt;=MaxTSSize), TBSize is derived in accordance with the expression (26) or (27) given hereinabove. Further, in the expression (26) or (27), TBSize, TBWSize and TBHSize may be replaced by logarithms log 2TBSize, log 2TBWSize and log 2TBHSize with the base 2. In this case, the expression (26) is replaced by the expression (28) given hereinabove and the expression (27) is replaced by the expression (29) given hereinabove. It is to be noted that, in the case where logarithms are used, TBSize and the maximum transform skip block size MaxTSSize of the conditional expression (TBSize&lt;=MaxTSSize) given hereinabove are replaced by corresponding logarithms log 2TBSize and log 2MaxTSSize, respectively. 
     In the case where it is decided that TBSize is greater than the maximum transform skip block size MaxTSSize, namely, in the case where the logical value of the conditional expression is 0, encoding of the transform skip identifier ts_idx is omitted and the encoding process is ended, and the processing returns to  FIG. 29 . On the other hand, in the case where it is decided that TBSize is equal to or smaller than the maximum transform skip block size MaxTSSize, namely, in the case where the logical value of the conditional expression is 1, the processing advances to step S 378 . 
     At step S 378 , the transform skip identifier encoding section  364  variable length encodes the transform skip identifier ts_idx included in the header information HInfo to generate a bit string and outputs the bit string. When the process at step S 378  ends, the encoding process ends, and the processing returns to  FIG. 29 . 
     In the foregoing, rearrangement of the processing order of the steps or change of the substance of the processes may be performed as far as practicable. 
     By executing the processes in such a manner as described above, on the encoding side, two-dimensional transform skip or one-dimensional transform skip in the horizontal direction or the vertical direction can be adaptively selected in a transform block unit in comparison with JEM3. Accordingly, since a residual signal in regard to which one-dimensional transform skip is more effective than before can be encoded by the mode of one-dimensional transform skip, the encoding efficiency can be enhanced. 
     &lt;Encoding of Primary Transform Identifier Pt_Idx&gt; 
       FIG. 34  is a block diagram depicting a principal configuration example relating to encoding of the primary transform identifier pt_idx of the encoding section  315 . As depicted in  FIG. 34 , the encoding section  315  includes a primary transform validity flag encoding section  371 , an adaptive primary transform flag encoding section  372  and a primary transform identifier encoding section  373 . 
     The primary transform validity flag encoding section  371  performs a process relating to encoding of the primary transform validity flag pt_enabled_flag. The adaptive primary transform flag encoding section  372  performs a process relating to encoding of the primary transform flag pt_enabled_flag. The primary transform identifier encoding section  373  performs a process relating to encoding of the primary transform identifier pt_idx. 
     An example of a flow of the encoding process relating to the primary transform identifier pt_idx, which is executed at step S 310  of  FIG. 29  by such a encoding section  315  as described above, is described with reference to a flow chart of  FIG. 35 . 
     After the encoding process is started, at step S 391 , the primary transform validity flag encoding section  371  variable length encodes the primary transform validity flag pt_enabled_flag included in the header information HInfo to generate a bit string and outputs the bit string. 
     At step S 392 , the adaptive primary transform flag encoding section  372  decides whether or not the primary transform validity flag pt_enabled_flag included in the header information Hinfo is 1 (true). In the case where it is decided that the primary transform validity flag pt_enabled_flag is 0, processes relating to derivation and encoding of the primary transform identifier pt_idx are omitted, and the encoding process ends and the processing returns to  FIG. 29 . On the other hand, in the case where it is decided that the primary transform validity flag pt_enabled_flag is 1, the processing advances to step S 393 . 
     At step S 393 , the adaptive primary transform flag encoding section  372  variable length encodes the adaptive primary transform flag apt_flag included in the header information HInfo to generate a bit string and outputs the bit string. 
     At step S 394 , the adaptive primary transform flag encoding section  372  decides whether or not the adaptive primary transform flag apt_flag is 1 (true). In the case where it is decided that the adaptive primary transform flag apt_flag is 0 (false), processes relating to derivation and encoding of the primary transform identifier pt_idx are omitted and the encoding process is ended, and the processing returns to  FIG. 29 . On the other hand, in the case where it is decided that the adaptive primary transform flag apt_flag is 1 (true), the processing advances to step S 395 . 
     At step S 395 , the primary transform identifier encoding section  373  decides whether or not the transform quantization bypass flag transquant_bypass_flag is 1 (true). In the case where it is decided that the transform quantization bypass flag transquant_bypass_flag is 1 (true), processes relating derivation and encoding of the primary transform identifier pt_idx are omitted and the encoding process is ended, and the processing returns to  FIG. 29 . On the other hand, in the case where it is decided that the transform quantization bypass flag transquant_bypass_flag is 0 (false), the processing advances to step S 396 . 
     At step S 396 , the primary transform identifier encoding section  373  decides whether or not the transform skip identifier ts_idx is 2D TS (two-dimensional transform skip). In the case where it is decided that the transform skip identifier ts_idx is 2D_TS, processes relating to derivation and encoding of the primary transform identifier pt_idx are omitted and the encoding process is ended, and the processing returns to  FIG. 29 . On the other hand, in the case where it is decided that the transform skip identifier ts_idx is any other than 2D TS, the processing advances to step S 397 . 
     At step S 397 , the primary transform identifier encoding section  373  decides whether or not at least one of the vertical size or the horizontal size of the transform block is equal to or smaller than the maximum adaptive primary transform block size MaxPTSize (max(TBHSize, TBWSize)&lt;=MaxPTSize). In the case where it is decided that at least one of the vertical size or the horizontal size of the transform block is equal to or smaller than the maximum adaptive primary transform block size MaxPTSize, processes relating to derivation and encoding of the primary transform identifier pt_idx are omitted and the encoding process is ended, and the processing returns to  FIG. 29 . On the other hand, in the case where it is decided that the transform skip identifier ts_idx is any other than 2D TS, the processing advances to step S 397 . In the case where it is decided that both the vertical size and the horizontal size of the transform block are equal to or smaller than the maximum adaptive primary transform block size MaxPTSize, the processing advances to step S 398 . 
     At step S 398 , the primary transform identifier encoding section  398  decides whether or not the transform block of the encoding target is a luminance component. In the case where it is decided that the transform block is not a luminance component, processes relating to derivation and encoding of the primary transform identifier pt_idx are omitted and the encoding process is ended, and the processing returns to  FIG. 29 . On the other hand, in the case where it is decided that the transform block is a luminance component, the processing advances to step S 399 . 
     At step S 399 , the primary transform identifier encoding section  398  refers to the residual information Rinfo to derive a total number numSig (total number of sig_coeff_flag==1) of non-zero transform coefficients existing in the transform block. After the process at step S 399 , the processing advances to step S 400 . 
     At step S 400 , the primary transform identifier encoding section  398  decides whether or not the total number numSig of non-zero transform coefficients is equal to or greater than a predetermined threshold value ptNumSigTH (numSig&gt;=ptNumSigTH). In the case where it is decided that the total number numSig of non-zero transform coefficients is smaller than the predetermined threshold value ptNumSigTH, processes relating to derivation and encoding of the primary transform identifier pt_idx are omitted and the encoding process is ended, and the processing returns to  FIG. 29 . On the other hand, in the case where it is decided that the total number numSig of non-zero transform coefficients is equal to or greater than the predetermined threshold value ptNumSigTH, the processing advances to step S 401 . 
     At step S 401 , the primary transform identifier encoding section  373  refers to the transform skip identifier ts_idx, primary horizontal transform designation flag pt_hor_flag and primary vertical transform designation flag pt_ver_flag to derive a primary transform identifier pt_idx. It is to be noted that detailed derivation of the primary transform identifier pt_idx is hereinafter described. 
     At step S 402 , the primary transform identifier encoding section  373  variable length encodes the primary transform identifier pt_idx included in the header information HInfo to generate a bit string and outputs the bit string. When the process at step S 402  ends, the encoding process ends and the processing returns to  FIG. 29 . 
     Although the foregoing description is directed to processes relating to encoding of the primary transform identifier pt_idx, rearrangement of the processing order of the steps or change of the substance of the processes may be performed as far as practicable. 
     As described above, in the case where the transform skip identifier ts_idx is 1D_H_TS or 1D_V_TS, since the primary transform identifier pt_idx can be arithmetically encoded as a bin string of 1 bit, the bin string that becomes an encoding target can be reduced from that in an alternative case in which the primary transform identifier pt_idx is arithmetically encoded as a bin string of 2 bits. Accordingly, the processing amount relating to encoding of the primary transform identifier pt_idx can be reduced. Further, since the code amount can be reduced, also enhancement of the encoding efficiency can be implemented. 
     Now, an example of a flow of the derivation process of the primary transform identifier pt_idx, which is executed at step S 401  of  FIG. 35  by the encoding section  315 , is described with reference to a flow chart of  FIG. 36 . 
     After the primary transform identifier derivation process is started, at step S 421 , the primary transform identifier encoding section  373  decides whether or not the transform skip identifier ts_idx is NO_TS (not transform skip). In the case where it is decided that the transform skip identifier ts_idx is NO_TS, the processing advances to step S 422 . 
     At step S 422 , the primary transform identifier encoding section  373  derives a primary transform identifier pt_idx from the primary horizontal transform designation flag pt_hor_flag and the primary vertical transform flag pt_ver_flag in accordance with the expression (20) given hereinabove. When the process at step  422  ends, the primary transform identifier derivation process ends and the processing returns to  FIG. 35 . 
     On the other hand, in the case where it is decided at step S 421  that the transform skip identifier ts_idx is any other than NO_TS, the processing advances to step S 423 . 
     At step S 423 , the primary transform identifier encoding section  373  decides whether or not the transform skip identifier ts_idx is 2D TS (two-dimensional transform skip). In the case where it is decided that the transform skip identifier ts_idx is 2D_TS, the primary transform identifier derivation process ends, and the processing returns to  FIG. 35 . 
     On the other hand, in the case where it is decided at step S 423  that the transform skip identifier ts_idx is any other than 2D_TS, the processing advances to step S 424 . 
     At step S 424 , the primary transform identifier encoding section  373  decides whether or not the transform skip identifier ts_idx is 1D_H_TS (one-dimensional transform skip in the horizontal direction). In the case where it is decided that the transform skip identifier ts_idx is 1D_H_TS, the processing advances to step S 425 . 
     At step S 425 , the primary transform identifier encoding section  373  derives the primary transform identifier pt_idx as pt_idx=pt_ver_flag on the basis of the expression (22) given hereinabove. When the process at step S 425  ends, the primary transform identifier derivation process ends, and the processing returns to  FIG. 35 . 
     On the other hand, in the case where it is decided at step S 424  that the transform skip identifier ts_idx is any other than 1D_H_TS, the processing advances to step S 426 . 
     At step S 426 , the primary transform identifier encoding section  373  derives the primary transform identifier pt_idx as pt_idx=pt_hor_flag on the basis of the expression (24) given hereinabove. When the process at step S 426  ends, the primary transform identifier derivation process ends, and the processing returns to  FIG. 35 . 
     Naturally, also this process may be subjected to rearrangement of the processing order of the steps or change of the substance of the processes as far as practicable. 
     &lt;Decoding of Secondary Transform Identifier st_idx&gt; 
       FIG. 37  is a block diagram depicting a principal configuration example relating to decoding of the secondary transform identifier st_idx of the encoding section  315 . As depicted in  FIG. 37 , the encoding section  315  includes a secondary transform validity flag encoding section  381  and a secondary transform identifier encoding section  382 . 
     The secondary transform validity flag encoding section  381  performs a process relating to encoding of the secondary transform validity flag st_enabled_flag. The secondary transform identifier encoding section  382  performs a process relating to encoding of the secondary transform identifier st_idx. 
     An example of a flow of the decoding process relating to the secondary transform identifier st_idx, which is executed at step S 310  of  FIG. 29  by the encoding section  315 , is described with reference to a flow chart of  FIG. 38 . 
     After the encoding process is started, at step S 441 , the secondary transform validity flag encoding section  381  variable length encodes the secondary transform validity flag st_enabled_flag included in the header information HInfo to generate a bit string and outputs the bit string. 
     At step S 442 , the secondary transform identifier encoding section  382  decides whether or not the secondary transform validity flag st_enabled_flag included in the header information Hinfo is 1 (true). In the case where it is decided that the secondary transform validity flag st_enabled_flag is 0 (false), the encoding process ends and the processing returns to  FIG. 29 . 
     On the other hand, in the case where it is decided at step S 442  that the secondary transform validity flag st_enabled_flag is 1 (true), the processing advances to step S 443 . 
     At step S 443 , the secondary transform identifier encoding section  382  decides whether or not the transform quantization bypass flag transquant_bypass_flag is 1 (true). In the case where it is decided that the transform quantization bypass flag transquant_bypass_flag is 1 (true), the encoding process is ended and the processing returns to  FIG. 29 . On the other hand, in the case where it is decided that the transform quantization bypass flag transquant_bypass_flag is 0 (false), the processing advances to step S 444 . 
     At step S 444 , the secondary transform identifier encoding section  382  decides whether or not the transform skip identifier ts_idx is NO_TS (transform skip is not to be performed). In the case where it is decided that the transform skip identifier ts_idx is any other than NO_TS, the encoding process is ended and the processing returns to  FIG. 29 . On the other hand, in the case where it is decided that the transform skip identifier ts_idx is NO_TS, the processing advances to step S 445 . 
     At step S 445 , the secondary transform identifier encoding section  382  refers to the residual information Rinfo to derive a total number numSig (total number of sig_coeff_flag==1) of non-zero transform coefficients existing in the transform block. 
     At step S 446 , the secondary transform identifier encoding section  382  decides whether or not the total number numSig of non-zero transform coefficients is equal to or greater than a predetermined threshold value TH (numSig&gt;=stNumSigTH). In the case where it is decided that the total number numSig of non-zero transform coefficients is smaller than the predetermined threshold value TH (numSig&lt;stNumSigTH), namely, that the logical value of the conditional expression is 0 (false), the secondary transform identifier encoding section  382  ends the encoding process, and the processing returns to  FIG. 29 . On the other hand, in the case where it is decided that the total number numSig of non-zero transform coefficients is equal to or greater than the predetermined threshold value TH (numSig&gt;=stNumSigTH), namely, that the logical value of the conditional expression is 1 (true), the processing advances to step S 447 . 
     At step S 447 , the secondary transform identifier encoding section  382  variable length encodes the secondary transform identifier st_idx included in the header information HInfo to generate a bit string and outputs the bit string. 
     When the process at step S 447  ends, the encoding process ends, and the processing returns to  FIG. 29 . 
     Also this process may be subjected to rearrangement of the processing order of the steps or change of the substance of the processes as far as practicable. Further, although the control parameter for a secondary transform in the foregoing description is the secondary transform identifier st_idx, it may otherwise be the secondary transform flag st_flag. 
     By the processes described above, in the case where a secondary transform identifier is to be encoded in a transform block unit, the encoding process of a secondary transform identifier can be omitted in the case where two-dimensional transform skip or one-dimensional transform skip is applied in the case of (ts_idx !=NO_TS) in comparison with the related art. In other words, the processing amount relating to encoding of a secondary transform identifier and the code amount can be reduced. 
     7. Seventh Embodiment 
     &lt;Encoding of Primary Horizontal Transform Designation Flag and Primary Vertical Transform Designation Flag&gt; 
     While it is described in the foregoing description that a primary transform identifier pt_idx is derived from a primary horizontal transform designation flag pt_hor_flag and a primary vertical transform designation flag pt_ver_flag, this is not restrictive. For example, in place of encoding the primary transform identifier pt_idx, the primary horizontal transform designation flag pt_hor_flag and the primary vertical transform designation flag pt_ver_flag may be encoded. 
       FIG. 39  is a block diagram depicting a principal configuration example relating to encoding of the primary horizontal transform designation flag pt_hor_flag and the primary vertical transform designation flag pt_ver_flag of the encoding section  315 . As depicted in  FIG. 39 , the encoding section  315  in this case includes a primary transform validity flag encoding section  371 , an adaptive primary transform flag encoding section  372 , a primary horizontal transform designation flag encoding section  391  and a primary vertical transform designation flag encoding section  392 . 
     The primary horizontal transform designation flag encoding section  391  performs a process relating to encoding of the primary horizontal transform designation flag pt_hor_flag. The primary vertical transform designation flag encoding section  392  performs a process relating to encoding of the primary vertical transform designation flag pt_ver_flag. 
     The primary horizontal transform designation flag encoding section  391  variable length encodes a primary horizontal transform designation flag pt_hor_flag from a bit string of encoded data #1 in the case where conditions of an if statement to which reference symbol SYN 15  is appended in the syntax depicted in  FIG. 25 , namely, in the case where the transform skip identifier ts_idx is NO_TS or 1D_V_TS (ts_idx==NO_TS II ts_idx==1D_V_TS); the transform quantization bypass flag is 0 (false) (transquant_bypass_flag==0); the transform block is a luminance component (cIdx==0); the adaptive primary transform flag apt_flag is 1 (true) (apt_flag==1); and besides the total number numSig of non-zero transform coefficients in the transform block is equal to or greater than the threshold value ptNumSigTH (numSig&gt;=ptNumSigTH). In any other case, the primary horizontal transform designation flag encoding section  391  omits encoding of the primary horizontal transform designation flag pt_hor_flag. 
     Similarly, the primary vertical transform designation flag encoding section  392  variable length encodes a primary vertical transform designation flag pt_ver_flag to generate a bit string in the case where conditions of an if statement to which reference symbol SYN 17  is appended in the syntax depicted in  FIG. 25 , namely, in the case where the transform skip identifier ts_idx is NO_TS or 1D_H_TS (ts_idx==NO_TS II ts_idx==1D_H_TS); the transform quantization bypass flag is 0 (false) (transquant_bypass_flag==0); the transform block is a luminance component (cIdx==0); the adaptive primary transform flag apt_flag is 1 (true) (apt_flag==1); and besides the total number numSig of non-zero transform coefficients in the transform block is equal to or greater than the threshold value ptNumSigTH (numSig&gt;=ptNumSigTH). In any other case, the primary vertical transform designation flag encoding section  392  omits encoding of the primary vertical transform designation flag pt_ver_flag. 
     By the foregoing, in the case where the transform skip identifier ts_idx is 1D_H_TS, encoding of the primary horizontal transform designation flag pt_hor_flag can be omitted. Similarly, in the case where the transform skip identifier ts_idx is 1D_V_TS, encoding of the primary vertical transform designation flag pt_ver_flag can be omitted. Accordingly, the processing amount relating to encoding of the primary horizontal transform designation flag pt_hor_flag and the primary vertical transform designation flag pt_ver_flag can be reduced. Further, since increase of the code amount can be suppressed, the encoding efficiency can be enhanced. 
     8. Eighth Embodiment 
     &lt;Other Configuration of Transform Section&gt; 
     Although it is described that, in the fourth embodiment, in the case where a primary horizontal transform is to be skipped, the switch  342  of the primary transform section  332  supplies a prediction residual D to the switch  344 , scaling for quantization may be performed thereupon. Further, although it is described that, in the fourth embodiment, in the case where a primary vertical transform is to be skipped, the switch  344  of the primary transform section  332  outputs transform coefficients Coeff_Phor after a primary horizontal transform as transform coefficients Coeff_P after a primary transform to the outside (secondary transform section  333 ), scaling for quantization may be performed thereupon. 
       FIG. 40  is a block diagram depicting a principal configuration example of the transform section  313  in this case. As depicted in  FIG. 40 , the transform section  313  also in this case has a basically similar configuration to that in the case of  FIG. 28 . It is to be noted, however, that, also in this case, the primary transform section  332  includes a scaling section  401  and another scaling section  402 . 
     In the case where a primary horizontal transform is to be skipped, the switch  342  supplies a transform coefficient D to the scaling section  401 . The scaling section  401  performs scaling for the transform coefficient D supplied from the switch  342 . For example, the scaling section  401  performs bit shift arithmetic operation of N (N is a natural number) bits for normalizing the transform coefficients supplied thereto such that they have a norm same as that in the case where a primary horizontal transform is carried out. The scaling section  401  supplies the scaled transform coefficients to the switch  344 . 
     In the case where a primary vertical transform is to be skipped, the switch  344  supplies transform coefficients Coeff_Phor after a primary horizontal transform as a prediction residual D′ to the scaling section  402 . The scaling section  402  performs scaling for the transform coefficients Coeff_Phor after a primary horizontal transform supplied from the switch  344  similarly as in the case of the scaling section  401 . For example, the scaling section  402  performs bit shift arithmetic operation of N (N is a natural number) bits for normalizing the transform coefficients supplied thereto such that they have a norm same as that in the case where a primary vertical transform is carried out. The scaling section  402  supplies the scaled prediction residual D′ to the outside. 
     Since this makes it possible to suppress the dynamic range width of the transform coefficients within a predetermined range, increase of the load of decoding can be suppressed. 
     9. Ninth Embodiment 
     &lt;Data Unit of Information&gt; 
     The data units to which information relating to an image or information relating to encoding or decoding of an image described hereinabove (or of data to be made a target) are arbitrary and are not limited to the examples described hereinabove. For example, such information may be set for each of a TU, a TB, a PU, a PB, a CU, an LCU, a sub block, a block, a tile, a slice, a picture, a sequence or a component, or data of such data units may be made a target. Naturally, the data units are set for each piece of information. In short, all information may not be set (or made a target) for each same data unit. It is to be noted that storage locations of such information are arbitrary, and the information may be stored for each header, each parameter set or the like of the data units described above. As an alternative, such information may be stored into a plurality of locations. 
     &lt;Control Information&gt; 
     Control information relating to the present technology described hereinabove in connection with the embodiments may be transmitted from the encoding side to the decoding side. For example, control information (for example, enabled_flag) for controlling whether or not it is to be permitted (or inhibited) to apply the present technology described hereinabove. Further, for example, control information that designates an upper limit or a lower limit or both of them to a block size to which application of the present technology described hereinabove is to be permitted (or inhibited) may be transmitted. 
     &lt;Encoding and Decoding&gt; 
     The present technology can be applied to arbitrary image encoding or decoding for performing a primary transform and a secondary transform (an inverse secondary transform and an inverse primary transform). In short, specifications for a transform (inverse transform), quantization (dequantization), encoding (decoding), prediction and so forth are arbitrary, and they are not limited to the examples described hereinabove. For example, in a transform (inverse transform), a (inverse) transform other than a (inverse) primary transform and a (inverse) secondary transform (namely, three or more (inverse) transforms) may be performed. Further, encoding (decoding) may be of the reversible type or of the irreversible type. Furthermore, quantization (dequantization), prediction and so forth may be omitted. Further, a process that has not been described such as a filter process may be performed. 
     &lt;Application Fields of Present Technology&gt; 
     Systems, apparatuses, processing sections and so forth to which the present technology is applied can be utilized in arbitrary fields such as, for example, transportation, medical care, crime prevention, agriculture, livestock industry, mining, beauty, factory, home appliances, weather, natural surveillance and so forth. 
     For example, the present technology can be applied also to systems and devices that transmit an image that is provided for viewing. Further, for example, the present technology can be applied also to systems and devices that are provided for transportation. Furthermore, for example, the present technology can be applied also to systems and devices that are provided for security use. Further, for example, the present technology can be applied also to systems and devices that are provided for sports. Furthermore, for example, the present technology can be applied also to systems and devices that are provided for agriculture. Further, for example, the present technology can be applied also to systems and devices that are provided for livestock industry. Furthermore, for example, the present technology can be applied also to systems and devices for monitoring conditions of the nature such as volcanoes, forests, oceans and so forth. Further, the present technology can be applied to a weather observation system or a weather observation apparatus for observing, for example, the weather, temperature, humidity, wind speed, sunshine hours and so forth. Furthermore, the present technology can be applied also to systems, devices and so forth for observing the ecology of wildlife such as, for example, birds, fish, reptiles, amphibians, mammals, insects, plants and so forth. 
     &lt;Application to Multi-View Image Encoding and Decoding System&gt; 
     The series of processes described hereinabove can be applied to a multi-view image encoding and decoding system that performs encoding and decoding of a multi-view image including images of a plurality of viewpoints (views (view)). In this case, it is sufficient if the present technology is applied to encoding or decoding of each viewpoint (view (view)). 
     &lt;Application to Hierarchical Image Encoding and Decoding System&gt; 
     Further, the series of processes described above can be applied to a hierarchical image encoding (scalable encoding) and decoding system that performs encoding and decoding of a hierarchical image layered (hierarchized) in a plurality of layers (hierarchies) so as to have a scalability (scalability) function in regard to a predetermined parameter. In this case, the present technology may be applied to encoding and decoding of each hierarchy (layer). 
     &lt;Computer&gt; 
     While the series of processes described above can be executed by hardware, it may otherwise be executed by software. In the case where the series of processes is executed by software, a program that constructs the software is installed into a computer. Here, the computer includes a computer incorporated in hardware for exclusive use, for example, a personal computer for universal use that can execute various functions by installing various programs, and so forth. 
       FIG. 41  is a block diagram depicting a configuration example of hardware of a computer that executes the series of processes described hereinabove in accordance with a program. 
     In the computer  800  depicted in  FIG. 41 , a CPU (Central Processing Unit)  801 , a ROM (Read Only Memory)  802  and a RAM (Random Access Memory)  803  are connected to each other by a bus  804 . 
     To the bus  804 , also an input/output interface  810  is connected. To the input/output interface  810 , an inputting section  811 , an outputting section  812 , a storage section  813 , a communication section  814  and a drive  815  are connected. 
     The inputting section  811  includes, for example, a keyboard, a mouse, a microphone, a touch panel, input terminals and so forth. The outputting section  812  includes, for example, a display, a speaker, output terminals and so forth. The storage section  813  includes, for example, a hard disk, a RAM disk, a nonvolatile memory and so forth. The communication section  814  includes, for example, a network interface or the like. The drive  815  drives a removable medium  821  such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory or the like. 
     In the computer configured in such a manner as described above, the CPU  801  loads a program stored, for example, in the storage section  813  into the RAM  803  through the input/output interface  810  and the bus  804  to perform the series of processes described above. Into the RAM  803 , also data and so forth necessary when the CPU  801  executes various processes are suitably stored. 
     The program that is executed by the computer (CPU  801 ) can be recorded into and applied to the removable medium  821 , for example, as a package medium or the like. In this case, the program can be installed into the storage section  813  through the input/output interface  810  by mounting the removable medium  821  on the drive  815 . 
     Further, the program can be provided through a wired or wireless transmission medium such as a local area network, the Internet, a digital satellite broadcast or the like. In this case, the program can be received by the communication section  814  and installed into the storage section  813 . 
     Further, the program can be installed in advance into the ROM  802  or the storage section  813 . 
     &lt;Application of Present Technology&gt; 
     The image encoding apparatus  300  and the image decoding apparatus  100  according to the embodiments described above can be applied to various electronic apparatus such as, for example, a transmitter or a receiver in satellite broadcasting, cable broadcasting of a cable TV or the like, delivery on the Internet, delivery to terminals by cellular communication, or a recording apparatus for recording an image on a medium such as an optical disk, a magnetic disk, a flash memory and so forth, a reproduction apparatus that reproduces from an image from such storage media, and so forth. 
     First Application Example: Television Receiver 
       FIG. 42  depicts an example of a schematic configuration of a television apparatus to which the embodiments described hereinabove are applied. The television apparatus  900  includes an antenna  901 , a tuner  902 , a demultiplexer  903 , a decoder  904 , a video signal processing section  905 , a display section  906 , an audio signal processing section  907 , a speaker  908 , an external interface (I/F) section  909 , a control section  910 , a user interface (I/F) section  911  and a bus  912 . 
     The tuner  902  extracts a signal of a desired channel from broadcasting signals received through the antenna  901  and demodulates the extracted signal. Then, the tuner  902  outputs an encoded bit stream obtained by the demodulation to the demultiplexer  903 . In other words, the tuner  902  has a role as a transmission section in the television apparatus  900 , which receives an encoded stream in which images are encoded. 
     The demultiplexer  903  demultiplexes a video stream and an audio stream of a broadcasting program of a viewing target from an encoded bit stream and outputs demultiplexed streams to the decoder  904 . Further, the demultiplexer  903  extracts auxiliary data such as an EPG (Electronic Program Guide) from the encoded bit stream and supplies the extracted data to the control section  910 . It is to be noted that, in the case where the encoded bit stream is in a scrambled state, the demultiplexer  903  may perform descrambling. 
     The decoder  904  decodes a video stream and an audio stream inputted from the demultiplexer  903 . Then, the decoder  904  outputs video data generated by the decoding process to the video signal processing section  905 . Further, the decoder  904  outputs audio data generated by the decoding process to the audio signal processing section  907 . 
     The video signal processing section  905  reproduces video data inputted from the decoder  904  and causes the display section  906  to display a video. Further, the video signal processing section  905  may cause the display section  906  to display an application screen image supplied thereto through a network. Further, the video signal processing section  905  may perform an additional process such as, for example, noise removal from video data in response to settings. Furthermore, the video signal processing section  905  may generate an image of a GUI (Graphical User Interface) such as, for example, a menu, a button or a cursor and superpose the generated image on an output image. 
     The display section  906  is driven by a driving signal supplied from the video signal processing section  905  and displays a video or an image on a video face of a display device (for example, a liquid crystal display, a plasma display or an OELD (Organic Electro Luminescence Display) (organic EL display) or the like). 
     The audio signal processing section  907  performs a reproduction process such as D/A conversion and amplification for audio data inputted from the decoder  904  and causes the speaker  908  to output sound. Further, the audio signal processing section  907  may perform an additional process such as noise removal for the audio data. 
     The external interface section  909  is an interface for connecting the television apparatus  900  and an external apparatus or a network to each other. For example, a video stream or an audio stream received through the external interface section  909  may be decoded by the decoder  904 . In particular, also the external interface section  909  has a role as a transmission section in the television apparatus  900 , which receives an encoded stream in which an image is encoded. 
     The control section  910  includes a processor such as a CPU, and a memory such as a RAM and a ROM. The memory stores therein a program to be executed by the CPU, program data, EPG data, data acquired through a network and so forth. The program stored in the memory is read by the CPU and executed, for example, upon activation of the television apparatus  900 . The CPU executes the program to control operation of the television apparatus  900 , for example, in response to an operation signal inputted from the user interface section  911 . 
     The user interface section  911  is connected to the control section  910 . The user interface section  911  includes, for example, buttons and switches for allowing a user to operate the television apparatus  900 , a reception section for a remote controlling signal and so forth. The user interface section  911  detects an operation by a user through the components mentioned to generate an operation signal and outputs the generated operation signal to the control section  910 . 
     The bus  912  connects the tuner  902 , demultiplexer  903 , decoder  904 , video signal processing section  905 , audio signal processing section  907 , external interface section  909  and control section  910  to each other. 
     In the television apparatus  900  configured in such a manner as described above, the decoder  904  may have the functions of the image decoding apparatus  100  described hereinabove. In short, the decoder  904  may decode encoded data by the methods described in the foregoing description of the embodiments. This makes it possible for the television apparatus  900  to achieve advantageous effects similar to those of the embodiments described hereinabove with reference to  FIGS. 1 to 40 . 
     Further, the television apparatus  900  configured in such a manner as described above may be configured such that the video signal processing section  905  can encode image data supplied, for example, from the decoder  904  and output resulting encoded data to the outside of the television apparatus  900  through the external interface section  909 . Further, the video signal processing section  905  may have the functions of the image encoding apparatus  300  described hereinabove. In short, the video signal processing section  905  may encode image data supplied from the decoder  904  by the methods described hereinabove in connection with the embodiments described hereinabove. This makes it possible for the television apparatus  900  to achieve advantageous effects similar to those of the embodiments described hereinabove with reference to  FIGS. 1 to 40 . 
     Second Application Example: Portable Telephone Set 
       FIG. 43  depicts an example of a schematic configuration of a portable telephone set to which the embodiments described hereinabove are applied. The portable telephone set  920  includes an antenna  921 , a communication section  922 , an audio codec  923 , a speaker  924 , a microphone  925 , a camera section  926 , an image processing section  927 , a demultiplexing section  928 , a recording and reproduction section  929 , a display section  930 , a control section  931 , an operation section  932  and a bus  933 . 
     The antenna  921  is connected to the communication section  922 . The speaker  924  and the microphone  925  are connected to the audio codec  923 . The operation section  932  is connected to the control section  931 . The bus  933  connects the communication section  922 , audio codec  923 , camera section  926 , image processing section  927 , demultiplexing section  928 , recording and reproduction section  929 , display section  930  and control section  931  to each other. 
     The portable telephone set  920  performs various operations such as transmission and reception of a voice signal, transmission and reception of an electronic mail or image data, imaging of an image, recording of data and so forth in various operation modes including a speech mode, a data communication mode, an imaging mode and a videophone mode. 
     In the speech mode, an analog voice signal generated by the microphone  925  is supplied to the audio codec  923 . The audio codec  923  converts the analog voice signal into voice data and A/D converts and compresses the voice data after the conversion. Then, the audio codec  923  outputs the compressed voice data to the communication section  922 . The communication section  922  encodes and modulates the voice data to generate a transmission signal. Then, the communication section  922  transmits the generated transmission signal to a base station (not depicted) through the antenna  921 . Further, the communication section  922  amplifies and frequency converts a wireless signal received through the antenna  921  to acquire a reception signal. Then, the communication section  922  demodulates and decodes the reception signal to generate voice data and outputs the generated voice data to the audio codec  923 . The audio codec  923  decompresses and D/A converts the voice data to generate an analog voice signal. Then, the audio codec  923  supplies the generated voice signal to the speaker  924  such that voice is outputted. 
     On the other hand, in the data communication mode, for example, the control section  931  generates character data that configure an electronic mail in response to an operation by a user through the operation section  932 . Further, the control section  931  controls the display section  930  to display characters thereon. Further, the control section  931  generates electronic mail data in response to a transmission instruction from the user through the operation section  932  and outputs the generated electronic mail data to the communication section  922 . The communication section  922  encodes and modulates the electronic mail data to generate a transmission signal. Then, the communication section  922  transmits the generated transmission signal to a base station (not depicted) through the antenna  921 . Further, the communication section  922  amplifies and frequency converts a wireless signal received through the antenna  921  to acquire a reception signal. Then, the communication section  922  demodulates and decodes the reception signal to restore electronic mail data and outputs the restored electronic mail data to the control section  931 . The control section  931  controls the display section  930  to display the substance of the electronic mail and supplies the electronic mail data to the recording and reproduction section  929  such that the electronic mail data is written into a storage medium of the recording and reproduction section  929 . 
     The recording and reproduction section  929  has an arbitrary readable/writable storage medium. For example, the storage medium may be a built-in type storage medium such as a RAM or a flash memory or may be an externally mounted storage medium such as a hard disk, a magnetic disk, a magneto-optical disk, an optical disk, a USB (Universal Serial Bus) memory or a memory card. 
     Further, in the imaging mode, for example, the camera section  926  images an image pickup object to generate image data and outputs the generated image data to the image processing section  927 . The image processing section  927  encodes the image data inputted from the camera section  926  and supplies an encoded stream to the recording and reproduction section  929  such that it is written into a storage medium of the recording and reproduction section  929 . 
     Furthermore, in the image display mode, the recording and reproduction section  929  reads out an encoded stream recorded in a storage medium and outputs the encoded stream to the image processing section  927 . The image processing section  927  decodes the encoded stream inputted from the recording and reproduction section  929  and supplies image data to the display section  930  such that an image of the image data is displayed. 
     Further, in the videophone mode, for example, the demultiplexing section  928  multiplexes a video stream encoded by the image processing section  927  and an audio stream inputted from the audio codec  923  and outputs the multiplexed stream to the communication section  922 . The communication section  922  encodes and modulates the stream to generate a transmission signal. Then, the communication section  922  transmits the generated transmission signal to a base station (not depicted) through the antenna  921 . Meanwhile, the communication section  922  amplifies and frequency converts a wireless signal received through the antenna  921  to acquire a reception signal. The transmission signal and the reception signal can include an encoded bit stream. Then, the communication section  922  demodulates and decodes the reception signal to restore a stream and outputs the restored stream to the demultiplexing section  928 . The demultiplexing section  928  demultiplexes a video stream and an audio stream from the inputted stream and outputs the video stream to the image processing section  927  while it outputs the audio stream to the audio codec  923 . The image processing section  927  decodes the video stream to generate video data. The video data is supplied to the display section  930 , by which a series of images are displayed. The audio codec  923  decompresses and D/A converts the audio stream to generate an analog voice signal. Then, the audio codec  923  supplies the generated voice signal to the speaker  924  such that voice is outputted from the speaker  924 . 
     In the portable telephone set  920  configured in this manner, for example, the image processing section  927  may have the functions of the image encoding apparatus  300  described hereinabove. In short, the image processing section  927  may encode image data by any of the methods described hereinabove in connection with the embodiments. This makes it possible for the portable telephone set  920  to achieve advantageous effects similar to those of the embodiments described hereinabove with reference to  FIGS. 1 to 40 . 
     Further, in the portable telephone set  920  configured in such a manner as described above, for example, the image processing section  927  may have the functions of the image decoding apparatus  100  described hereinabove. In short, the image processing section  927  may decode encoded data by any of the methods described hereinabove in connection with the embodiments. This makes it possible for the portable telephone set  920  to achieve advantageous effects similar to those of the embodiments described hereinabove with reference to  FIGS. 1 to 40 . 
     Third Application Example: Recording and Reproduction Apparatus 
       FIG. 44  depicts an example of a schematic configuration of a recording and reproduction apparatus to which the embodiments described hereinabove are applied. The recording and reproduction apparatus  940  encodes and records, for example, audio data and video data of a received broadcasting program into a recording medium. Further, the recording and reproduction apparatus  940  may encode and record audio data and video data acquired, for example, from a different apparatus into a recording medium. Further, the recording and reproduction apparatus  940  reproduces data recorded in the recording medium on a monitor and a speaker, for example, in response to an instruction of a user. At this time, the recording and reproduction apparatus  940  decodes audio data and video data. 
     The recording and reproduction apparatus  940  includes a tuner  941 , an external interface (I/F) section  942 , an encoder  943 , an HDD (Hard Disk Drive) section  944 , a disk drive  945 , a selector  946 , a decoder  947 , an OSD (On-Screen Display) section  948 , a control section  949 , and a user interface (I/F) section  950 . 
     The tuner  941  extracts a signal of a desired channel from broadcasting signals received through an antenna (not depicted) and demodulates the extracted signal. Then, the tuner  941  outputs an encoded bit stream obtained by the demodulation to the selector  946 . In other words, the tuner  941  has a role as a transmission section in the recording and reproduction apparatus  940 . 
     The external interface section  942  is an interface for connecting the recording and reproduction apparatus  940  and an external apparatus or a network to each other. The external interface section  942  may be, for example, an IEEE (Institute of Electrical and Electronic Engineers) 1394 interface, a network interface, a USB interface, a flash memory interface or the like. For example, video data and audio data received through the external interface section  942  are inputted to the encoder  943 . In other words, the external interface section  942  has a role as a transmission section in the recording and reproduction apparatus  940 . 
     The encoder  943  encodes video data and audio data inputted form the external interface section  942  in the case where the video data and the audio data are not in an encoded form. Then, the encoder  943  outputs an encoded bit stream to the selector  946 . 
     The HDD section  944  records an encoded bit stream, in which content data of videos and audios are compressed, various programs and other data on an internal hard disk thereof. Further, upon reproduction of videos and audios, the HDD section  944  reads out such data from the hard disk. 
     The disk drive  945  performs recording and reading out of data on and from a recording medium loaded therein. The recording medium to be loaded into the disk drive  945  may be, for example, a DVD (Digital Versatile Disc) disk (DVD-Video, DVD-RAM (DVD-Random Access Memory), DVD-R (DVD-Readable), DVD-RW (DVD-Rewritable), DVD+R (DVD+Recordable), DVD+RW (DVD+Rewritable) and so forth) or a Blu-ray (registered trademark) disk or the like. 
     Upon recording of videos and audios, the selector  946  selects an encoded bit stream inputted from the tuner  941  or the encoder  943  and outputs the selected encoded bit stream to the HDD section  944  or the disk drive  945 . On the other hand, upon reproduction of videos and audios, the selector  946  outputs an encoded bit stream inputted from the HDD section  944  or the disk drive  945  to the decoder  947 . 
     The decoder  947  decodes an encoded bit stream to generate video data and audio data. Then, the decoder  947  outputs the generated video data to the OSD section  948 . Meanwhile, the decoder  947  outputs the generated audio data to an external speaker. 
     The OSD section  948  reproduces video data inputted from the decoder  947  to display a video. Further, the OSD section  948  may superimpose an image of a GUI such as, for example, a menu, a button or a cursor on the video to be displayed. 
     The control section  949  includes a processor such as a CPU and a memory such as a RAM and a ROM. The memory stores therein a program to be executed by the CPU, program data and so forth. The program stored in the memory is read into and executed by the CPU, for example, upon activation of the recording and reproduction apparatus  940 . The CPU executes the program to control operation of the recording and reproduction apparatus  940  in response to an operation signal inputted, for example, from the user interface section  950 . 
     The user interface section  950  is connected to the control section  949 . The user interface section  950  has buttons and switches for allowing, for example, a user to operate the recording and reproduction apparatus  940  and a reception section for a remote controlling signal and so forth. The user interface section  950  detects an operation by the user through the components mentioned to generate an operation signal and outputs the generated operation signal to the control section  949 . 
     In the recording and reproduction apparatus  940  configured in this manner, for example, the encoder  943  may have the functions of the image encoding apparatus  300  described hereinabove. In short, the encoder  943  may encode image data by a method described in connection with the embodiments. This makes it possible for the recording and reproduction apparatus  940  to achieve advantageous effects similar to those of the embodiments described hereinabove with reference to  FIGS. 1 to 40 . 
     Further, in the recording and reproduction apparatus  940  configured in this manner, for example, the decoder  947  may have the functions of the image decoding apparatus  100  described hereinabove. In short, the decoder  947  may decode encoded data by any method described in the foregoing description of the embodiments. This makes it possible for the recording and reproduction apparatus  940  to achieve advantageous effects similar to those of the embodiments described hereinabove with reference to  FIGS. 1 to 40 . 
     Fourth Application Example: Imaging Apparatus 
       FIG. 45  depicts an example of a schematic configuration of an imaging apparatus to which the embodiments described above are applied. The imaging apparatus  960  images an image pickup object to generate an image and encodes and records image data into a recording medium. 
     The imaging apparatus  960  includes an optical block  961 , an imaging section  962 , a signal processing section  963 , an image processing section  964 , a display section  965 , an external interface (I/F) section  966 , a memory section  967 , a media drive  968 , an OSD section  969 , a control section  970 , a user interface (I/F) section  971  and a bus  972 . 
     The optical block  961  is connected to the imaging section  962 . The imaging section  962  is connected to the signal processing section  963 . The display section  965  is connected to the image processing section  964 . The user interface section  971  is connected to the control section  970 . The bus  972  connects the image processing section  964 , external interface section  966 , memory section  967 , media drive  968 , OSD section  969  and control section  970  to each other. 
     The optical block  961  has a focus lens, a diaphragm mechanism and so forth. The optical block  961  forms an optical image of an image pickup object on an imaging face of the imaging section  962 . The imaging section  962  includes an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor and converts the optical image formed on the imaging face into an image signal as an electric signal by photoelectric conversion. Then, the imaging section  962  outputs the image signal to the signal processing section  963 . 
     The signal processing section  963  performs various camera signal processes such as knee correction, gamma correction and color correction for the image signal inputted from the imaging section  962 . The signal processing section  963  outputs image data after the camera signal processes to the image processing section  964 . 
     The image processing section  964  encodes the image data inputted from the signal processing section  963  to generate encoded data. Then, the image processing section  964  outputs the generated encoded data to the external interface section  966  or the media drive  968 . Further, the image processing section  964  decodes encoded data inputted from the external interface section  966  or the media drive  968  to generate image data. Then, the image processing section  964  outputs the generated image data to the display section  965 . Further, the image processing section  964  may output the image data inputted from the signal processing section  963  to the display section  965  such that an image is displayed on the display section  965 . Further, the image processing section  964  may superimpose displaying data acquired from the OSD section  969  on the image to be outputted to the display section  965 . 
     The OSD section  969  generates an image of a GUI such as, for example, a menu, a button or a cursor and outputs the generated image to the image processing section  964 . 
     The external interface section  966  is configured, for example, as USB input/output terminals. The external interface section  966  connects the imaging apparatus  960  and a printer to each other, for example, upon printing of the image. Further, a drive is connected to the external interface section  966  as occasion demands. A removable medium such as, for example, a magnetic disk or an optical disk is mounted on the drive, and a program read out from the removable medium can be installed into the imaging apparatus  960 . Further, the external interface section  966  may be configured as a network interface that is connected to a network such as a LAN or the Internet. In particular, the external interface section  966  has a role as a transmission section in the imaging apparatus  960 . 
     The recording medium to be mounted on the media drive  968  may be an arbitrary rewritable removable medium such as, for example, a magnetic disk, a magneto-optical disk, an optical disk or a semiconductor memory. Further, the recording medium may be fixedly mounted on the media drive  968  such that a non-portable storage section like, for example, a built-in hard disk drive or an SSD (Solid State Drive) is configured. 
     The control section  970  includes a processor such as a CPU and a memory such as a RAM and a RAM. The memory has a program to be executed by the CPU, program data and so forth stored therein. The program stored in the memory is read into and executed by the CPU upon activation of the imaging apparatus  960 . The CPU executes the program to control operation of the imaging apparatus  960  in accordance with, for example, an operation signal inputted from the user interface section  971 . 
     The user interface section  971  is connected to the control section  970 . The user interface section  971  has, for example, buttons, switches and so forth for operation of the imaging apparatus  960  by the user. The user interface section  971  detects an operation by the user through the components described above to generate an operation signal and outputs the generated operation signal to the control section  970 . 
     In the imaging apparatus  960  configured in such a manner as described above, for example, the image processing section  964  may have the functions of the image encoding apparatus  300  described above. In particular, the image processing section  964  may encode image data by any method described in connection with the embodiments described above. This makes it possible for the imaging apparatus  960  to achieve advantageous effects similar to those of the embodiments described hereinabove with reference to  FIGS. 1 to 40 . 
     Further, in the imaging apparatus  960  configured in such a manner as described above, for example, the image processing section  964  may have the functions of the image decoding apparatus  100  described hereinabove. In particular, the image processing section  964  may decode encoded data by any method described in connection with the embodiments described above. This makes it possible for the imaging apparatus  960  to achieve advantageous effects similar to those of the embodiments described hereinabove with reference to  FIGS. 1 to 40 . 
     Fifth Application Example: Video Set 
     Further, the present technology can be carried out as any configuration to be incorporated in an arbitrary apparatus or an apparatus configuring a system such as, for example, a processor as a system LSI (Large Scale Integration) or the like, a module in which a plurality of processors or the like are used, a unit in which a plurality of modules are used, a set in which a different function is further added to the unit (namely, part of the configuration of the apparatus).  FIG. 46  depicts an example of a schematic configuration of a video set to which the present technology is applied. 
     In recent years, multifunctionalization of electronic equipment has been and is being advanced, and, in the case where some component is carried out as selling, provision or the like in development or fabrication of the electronic equipment, not only a case in which the component is carried out as a component having one function but also a case in which a plurality of components having functions relating to each other are combined and carried out as one set having a plurality of functions are seen frequently. 
     A video set  1300  depicted in  FIG. 46  has such a multifunctionalized configuration as described above and is an apparatus in which a device having functions relating to encoding and decoding of an image (one of or both encoding and decoding may be applied) and another device having other functions relating to the functions are combined. 
     As depicted in  FIG. 46 , the video set  1300  includes a module group including a video module  1311 , an external memory  1312 , a power management module  1313  and a front-end module  1314  and a device having relating functions such as a connectivity  1321 , a camera  1322 , a sensor  1323  and so forth. 
     A module is a part in which several part functions relating to each other are combined so as to have coherent functions. Although a particular physical configuration is arbitrary, a physical configuration is conceivable in which, for example, a plurality of processors individually having functions, electronic circuit devices such as resisters and capacitors, other devices and so forth are disposed and integrated on a wiring board or the like. Also it is conceivable to combine a different module, a processor or the like with a module to produce a new module. 
     In the case of the example of  FIG. 46 , the video module  1311  is a combination of components having functions relating to image processing and includes an application processor, a video processor, a broadband modem  1333  and an RF module  1334 . 
     The processor is an integration of a component having a predetermined function with a semiconductor chip by SoC (System On a Chip), and also a processor called, for example, as system LSI (Large Scale Integration) or the like is available. The component that has a predetermined function may be a logic circuit (hardware configuration) or may be a CPU, a ROM, a RAM and so forth and a program executed using them or else may be a combination of both of them. For example, the processor may include a logic circuit, a CPU, a ROM, a RAM and so forth such that part of functions are implemented by the logic circuit (hardware configuration) and the remaining part of the functions are implemented by the program to be executed by the CPU (software configuration). 
     The application processor  1331  of  FIG. 46  is a processor that executes an application relating to an image process. In order to implement a predetermined function, the application to be executed by the application processor  1331  not only can perform an arithmetic operation process but also can control, as occasion demands, the components of the inside and the outside of the video module  1311  such as, for example, the video processor  1332 . 
     The video processor  1332  is a processor having functions relating to (one of or both) encoding and decoding of an image. 
     The broadband modem  1333  converts data (digital signal) to be transmitted by wire or wireless (or both) broadband communication performed through a broadband line such as the Internet or a public telephone network into an analog signal by digital modulation of the data or the like or converts an analog signal received by the broadband communication into data (digital signal) by demodulation of the analog signal. The broadband modem  1333  processes arbitrary information such as, for example, image data to be processed by the video processor  1332 , a stream in which image data is encoded, an application program or setting data. 
     The RF module  1334  is a module that performs frequency conversion, modulation and demodulation, amplification, filter processing and so forth for an RF (Radio Frequency) signal sent and received through an antenna. For example, the RF module  1334  performs frequency conversion and so forth for a baseband signal generated by the broadband modem  1333  to generate an RF signal. Further, for example, the RF module  1334  performs frequency conversion and so forth for an RF signal received through the front-end module  1314  to generate a baseband signal. 
     It is to be noted that, as depicted by a broken line  1341  in  FIG. 46 , the application processor  1331  and the video processor  1332  may be integrated so as to be configured as one processor. 
     The external memory  1312  is a module that is provided on the outside of the video module  1311  and has a storage device to be utilized by the video module  1311 . While the storage device of the external memory  1312  may be implemented by any physical component, since generally the storage device is frequently utilized for storage of a great amount of data like image data of a frame unit, it is desirable to implement the storage device by a comparatively-low-price and great-capacity semiconductor memory such as, for example, a DRAM (Dynamic Random Access Memory). 
     The power management module  1313  manages and controls power supply to the video module  1311  (components in the video module  1311 ). 
     The front-end module  1314  is a module that provides a front-end function (circuit at a transmission or reception end on the antenna side) to the RF module  1334 . As depicted in  FIG. 46 , the front-end module  1314  includes, for example, an antenna section  1351 , a filter  1352  and an amplification section  1353 . 
     The antenna section  1351  includes an antenna for transmitting and receiving a wireless signal and peripheral components of the antenna. The antenna section  1351  transmits a signal supplied from the amplification section  1353  as a wireless signal and supplies a received wireless signal as an electric signal (RF signal) to the filter  1352 . The filter  1352  performs a filter process and so forth for the RF signal received through the antenna section  1351  and supplies the RF signal after the process to the RF module  1334 . The amplification section  1353  amplifies and supplies an RF signal supplied from the RF module  1334  to the antenna section  1351 . 
     The connectivity  1321  is a module having a function relating to connection to the outside. The physical configuration of the connectivity  1321  is arbitrary. For example, the connectivity  1321  has a component having a communication function of a communication standard, which is different from that with which the broadband modem  1333  is compatible, external input/output terminals and so forth. 
     For example, the connectivity  1321  may include a module having a communication function that complies with a wireless communication standard such as Bluetooth (registered trademark), IEEE 802.11 (for example, Wi-Fi (Wireless Fidelity, registered trademark)), NFC (Near Field Communication) or IrDA (InfraRed Data Association), an antenna for transmitting and receiving a signal that complies with the standard and so forth. Further, for example, the connectivity  1321  may include a module having a communication function that complies with a wire communication standard such as USB (Universal Serial Bus), HDMI (registered trademark) (High-Definition Multimedia Interface) or the like and a terminal that complies with the standard. Furthermore, for example, the connectivity  1321  may have a different data (signal) transmission function or the like such as analog input/output terminals or the like. 
     It is to be noted that the connectivity  1321  may include a device of a transmission destination of data (signal). For example, the connectivity  1321  may include a drive that performs reading out and writing of data from and into a recording medium such as a magnetic disk, an optical disk, a magneto-optical disk or a semiconductor memory (include not only a drive for a removable medium but also a drive for a hard disk, an SSD (Solid State Drive), an NAS (Network Attached Storage)) or the like. Further, the connectivity  1321  may include an outputting device for an image or sound (a monitor, a speaker or the like). 
     The camera  1322  is a module having a function for imaging an image of an imaging object to obtain image data of the imaging object. The image data obtained by imaging of the camera  1322  is supplied to and encoded by, for example, the video processor  1332 . 
     The sensor  1323  is a module having an arbitrary sensor function of, for example, a sound sensor, an ultrasonic sensor, an optical sensor, an illumination sensor, an infrared sensor, an image sensor, a rotation sensor, an angle sensor, an angular velocity sensor, a velocity sensor, an acceleration sensor, an inclination sensor, a magnetic identification sensor, a shock sensor, a temperature sensor or the like. Data detected by the sensor  1323  is supplied, for example, to the application processor  1331  and is utilized by an application or the like. 
     The components described as the modules in the foregoing description may be implemented as a processor, or conversely the component described as a processor may be implemented as a module. 
     In the video set  1300  having such a configuration as described above, the present technology can be applied to the video processor  1332  as hereinafter described. Accordingly, the video set  1300  can be carried out as a set to which the present technology is applied. 
     &lt;Configuration Example of Video Processor&gt; 
       FIG. 47  depicts an example of a schematic configuration of the video processor  1332  ( FIG. 46 ) to which the present technology is applied. 
     In the case of the example of  FIG. 47 , the video processor  1332  has a function for receiving an input of a video signal and an audio signal and encoding the signals by a predetermined method and another function for decoding video data and audio data in an encoded form and reproducing and outputting a video signal and an audio signal. 
     As depicted in  FIG. 47 , the video processor  1332  includes a video input processing section  1401 , a first image expansion/reduction section  1402 , a second image expansion/reduction section  1403 , a video output processing section  1404 , a frame memory  1405  and a memory controlling section  1406 . The video processor  1332  further includes an encode/decode engine  1407 , video ES (Elementary Stream) buffers  1408 A and  1408 B, and audio ES buffers  1409 A and  1409 B. The video processor  1332  further includes an audio encoder  1410 , an audio decoder  1411 , a multiplexing section (MUX (Multiplexer))  1412 , a demultiplexing section (DMUX (Demultiplexer))  1413 , and a stream buffer  1414 . 
     The video inputting processing section  1401  acquires a video signal inputted, for example, from the connectivity  1321  ( FIG. 46 ) or the like and converts the acquired video signal into digital image data. The first image expansion/reduction section  1402  performs format conversion, an expansion/reduction process of an image and so forth for the image data. The second image expansion/reduction section  1403  performs an expansion/reduction process of an image in accordance with a format at a destination of outputting through the video output processing section  1404  and performs format conversion, an expansion/reduction process of an image or the like similar to that of the first image expansion/reduction section  1402  for the image data. The video output processing section  1404  performs format conversion, conversion into an analog signal and so forth for the image data and outputs resulting data as a reproduced video signal, for example, to the connectivity  1321  and so forth. 
     The frame memory  1405  is a memory for image data shared by the video inputting processing section  1401 , first image expansion/reduction section  1402 , second image expansion/reduction section  1403 , video output processing section  1404  and encode/decode engine  1407 . The frame memory  1405  is implemented as a semiconductor memory such as, for example, a DRAM. 
     The memory controlling section  1406  receives a synchronizing signal from the encode/decode engine  1407  and controls accessing for writing and reading out to the frame memory  1405  in accordance with an access schedule to the frame memory  1405  written in an access management table  1406 A. The access management table  1406 A is updated by the memory controlling section  1406  in response to a process executed by the encode/decode engine  1407 , first image expansion/reduction section  1402 , second image expansion/reduction section  1403  or the like. 
     The encode/decode engine  1407  performs an encoding process of image data and a decoding process of a video stream that is data encoded from image data. For example, the encode/decode engine  1407  encodes image data read out from the frame memory  1405  and successively writes the encoded image data as a video stream into the video ES buffer  1408 A. Further, the encode/decode engine  1407  successively reads out a video stream, for example, from the video ES buffer  1408 B and decodes the video stream, and successively writes the decoded video stream as image data into the frame memory  1405 . The encode/decode engine  1407  uses the frame memory  1405  as a working area in the encoding and decoding. Further, the encode/decode engine  1407  outputs a synchronizing signal to the memory controlling section  1406 , for example, at a timing at which processing for each macro block is to be started. 
     The video ES buffer  1408 A buffers a video stream generated by the encode/decode engine  1407  and supplies the resulting video stream to the multiplexing section (MUX)  1412 . The video ES buffer  1408 B buffers a video stream supplied from the demultiplexing section (DMUX)  1413  and supplies the resulting video stream to the encode/decode engine  1407 . 
     The audio ES buffer  1409 A buffers an audio stream generated by the audio encoder  1410  and supplies the resulting audio stream to the multiplexing section (MUX)  1412 . The audio ES buffer  1409 B buffers an audio stream supplied from the demultiplexing section (DMUX)  1413  and supplies the resulting audio stream to the audio decoder  1411 . 
     The audio encoder  1410  performs, for example, digital conversion for an audio signal inputted, for example, from the connectivity  1321  or the like and encodes the resulting audio signal by a predetermined method such as, for example, an MPEG audio method or an AC 3  (AudioCode number 3) method. The audio encoder  1410  successively writes the audio stream that is data encoded from the audio signal into the audio ES buffer  1409 A. The audio decoder  1411  decodes an audio stream supplied from the audio ES buffer  1409 B and performs, for example, conversion into an analog signal or the like and then supplies the resulting analog signal as a reproduced audio signal, for example, to the connectivity  1321  and so forth. 
     The multiplexing section (MUX)  1412  multiplexes a video stream and an audio stream. The method of the multiplexing (namely, format of a bit stream generated by multiplexing) is arbitrary. Further, upon such multiplexing, the multiplexing section (MUX)  1412  can also add predetermined header information and so forth to the bit stream. In short, the multiplexing section (MUX)  1412  can convert the format of a stream by multiplexing. For example, the multiplexing section (MUX)  1412  multiplexes a video stream and an audio stream to convert them into a transport stream that is a bit stream of a transfer format. Further, for example, the multiplexing section (MUX)  1412  multiplexes the video stream and the audio stream to convert them into data of a recording file format (file data). 
     The demultiplexing section (DMUX)  1413  demultiplexes a bit stream, in which a video stream and an audio stream are multiplexed, by a method corresponding to the multiplexing by the multiplexing section (MUX)  1412 . In particular, the demultiplexing section (DMUX)  1413  extracts the video stream and the audio stream from the bit stream read out from the stream buffer  1414  (separates the video stream and the audio stream). In short, the demultiplexing section (DMUX)  1413  can convert the format of the stream by the demultiplexing (inverse conversion to the conversion by the multiplexing section (MUX)  1412 ). For example, the demultiplexing section (DMUX)  1413  can acquire a transport stream supplied, for example, from the connectivity  1321 , broadband modem  1333  or the like through the stream buffer  1414  and demultiplex the acquired stream so as to convert it into a video stream and an audio stream. Further, for example, the demultiplexing section (DMUX)  1413  can acquire file data read out from various recording media, for example, by the connectivity  1321  and can demultiplex the read out file data so as to convert it into a video stream and an audio stream. 
     The stream buffer  1414  buffers a bit stream. For example, the stream buffer  1414  buffers a transport stream supplied from the multiplexing section (MUX)  1412  and supplies the buffered transport stream, for example, to the connectivity  1321 , broadband modem  1333  or the like at a predetermined timing or on the basis of a request from the outside or the like. 
     Further, for example, the stream buffer  1414  buffers file data supplied from the multiplexing section (MUX)  1412  and supplies the buffered file data, for example, to the connectivity  1321  or the like so as to be recorded on various recording media at a predetermined timing or on the basis of a request from the outside or the like. 
     Furthermore, the stream buffer  1414  buffers a transport stream acquired, for example, through the connectivity  1321 , broadband modem  1333  or the like and supplies the buffered transport stream to the demultiplexing section (DMUX)  1413  at a predetermined timing or on the basis of a request from the outside or the like. 
     Further, the stream buffer  1414  buffers file data read out from various recording media, for example, by the connectivity  1321  or the like and supplies the buffered file data to the demultiplexing section (DMUX)  1413  at a predetermined timing or on the basis of a request from the outside or the like. 
     Now, an example of operation of the video processor  1332  having such a configuration as described above is described. For example, a video signal inputted from the connectivity  1321  or the like to the video processor  1332  is converted into digital image data of a predetermined method such as a 4:2:2Y/Cb/Cr method by the video inputting processing section  1401  and successively written into the frame memory  1405 . The digital image data is read out by the first image expansion/reduction section  1402  or the second image expansion/reduction section  1403 , subjected to format conversion into that of a predetermined method such as a 4:2:0Y/Cb/Cr method and an expansion/reduction process, and written back into the frame memory  1405 . This image data is encoded by the encode/decode engine  1407  and written as a video stream into the video ES buffer  1408 A. 
     Further, an audio signal inputted from the connectivity  1321  or the like to the video processor  1332  is encoded by the audio encoder  1410  and written as an audio stream into the audio ES buffer  1409 A. 
     The video stream of the video ES buffer  1408 A and the audio stream of the audio ES buffer  1409 A are read out to and multiplexed by the multiplexing section (MUX)  1412 , by which they are converted into a transport stream, file data or the like. The transport stream generated by the multiplexing section (MUX)  1412  is buffered by the stream buffer  1414  and then outputted to the external network, for example, through the connectivity  1321 , broadband modem  1333  and so forth. Further, the file data generated by the multiplexing section (MUX)  1412  is buffered by the stream buffer  1414 , whereafter it is outputted, for example, to the connectivity  1321  or the like and recorded on various recording media. 
     Further, a transport stream inputted from the external network to the video processor  1332 , for example, through the connectivity  1321 , broadband modem  1333  or the like is buffered by the stream buffer  1414  and then demultiplexed by the demultiplexing section (DMUX)  1413 . Further, file data read out from various recording media, for example, by the connectivity  1321  or the like and inputted to the video processor  1332  is buffered by the stream buffer  1414  and then demultiplexed by the demultiplexing section (DMUX)  1413 . In short, a transport stream or file data inputted to the video processor  1332  is separated into a video stream and an audio stream by the demultiplexing section (DMUX)  1413 . 
     The audio stream is supplied through the audio ES buffer  1409 B to and decoded by the audio decoder  1411  such that an audio signal is reproduced. Meanwhile, the video stream is written into the video ES buffer  1408 B, and then is successively read out and decoded by the encode/decode engine  1407  and written into the frame memory  1405 . The decoded image data is subjected to an expansion/reduction process by the second image expansion/reduction section  1403  and written into the frame memory  1405 . Then, the decoded image data is read out by the video output processing section  1404  and subjected to format conversion to a format of a predetermined method such as a 4:2:2Y/Cb/Cr method, whereafter it is converted further into an analog signal such that a video signal is reproduced and outputted. 
     In the case where the present technology is applied to the video processor  1332  configured as in this manner, it is sufficient if the present technology according to the embodiments described above is applied to the encode/decode engine  1407 . In particular, for example, the encode/decode engine  1407  may have the functions of the image encoding apparatus  300  or the functions of the image decoding apparatus  100  described above or both of them. This makes it possible for the video processor  1332  to achieve advantageous effects similar to those of the embodiments described above with reference to  FIGS. 1  to  40 . 
     It is to be noted that, in the encode/decode engine  1407 , the present technology (namely, the functions of the image encoding apparatus  300  or the functions of the image decoding apparatus  100  or both of them) may be implemented by hardware such as a logic circuit or may be implemented by software such as an embedded program, or may be implemented by both of them. 
     &lt;Different Configuration Example of Video Processor&gt; 
       FIG. 48  depicts another example of a schematic configuration of the video processor  1332  to which the present technology is applied. In the case of the example of  FIG. 48 , a video processor  1332  has a function for encoding and decoding video data by a predetermined method. 
     More particularly, as depicted in  FIG. 48 , the video processor  1332  includes a control section  1511 , a display interface  1512 , a display engine  1513 , an image processing engine  1514  and an internal memory  1515 . The video processor  1332  further includes a codec engine  1516 , a memory interface  1517 , a multiplexing and demultiplexing section (MUX DMUX)  1518 , a network interface  1519  and a video interface  1520 . 
     The control section  1511  controls operation of processing sections in the video processor  1332  such as the display interface  1512 , display engine  1513 , image processing engine  1514  and codec engine  1516 . 
     As depicted in  FIG. 48 , the control section  1511  includes, for example, a main CPU  1531 , a sub CPU  1532  and a system controller  1533 . The main CPU  1531  executes a program for controlling operation of the processing sections in the video processor  1332  and so forth. The main CPU  1531  generates control signals in accordance with the program and so forth and supplies the control signal to the processing sections (namely, controls operation of the processing sections). The sub CPU  1532  plays an auxiliary role for the main CPU  1531 . For example, the sub CPU  1532  executes a child process, a sub routine and so forth of the program and so forth to be executed by the main CPU  1531 . The system controller  1533  controls operation of the main CPU  1531  and the sub CPU  1532  such as designation of a program to be executed by the main CPU  1531  and the sub CPU  1532  or the like. 
     The display interface  1512  outputs image data, for example, to the connectivity  1321  or the like under the control of the control section  1511 . For example, the display interface  1512  converts image data of digital data into an analog signal and outputs the analog signal as a reproduced video signal or outputs the image data of digital data as it is to a monitor apparatus or the like of the connectivity  1321 . 
     The display engine  1513  performs various transform processes such as format conversion, size conversion and color gamut conversion for image data under the control of the control section  1511  so as to match with hardware specifications for a monitor apparatus for displaying an image and so forth. 
     The image processing engine  1514  performs a predetermined image process such as, for example, a filter process for picture quality improvement for the image data under the control of the control section  1511 . 
     The internal memory  1515  is a memory provided in the inside of the video processor  1332  and commonly used by the display engine  1513 , image processing engine  1514  and codec engine  1516 . The internal memory  1515  is utilized, for example, for sending and reception of data performed between the display engine  1513 , image processing engine  1514  and codec engine  1516 . For example, the internal memory  1515  stores data supplied from the display engine  1513 , image processing engine  1514  or codec engine  1516  and supplies the data to the display engine  1513 , image processing engine  1514  or codec engine  1516  as occasion demands (for example, in response to a request). While the internal memory  1515  may be implemented by any storage device, since generally the internal memory  1515  is frequently utilized for storage of a small amount of data such as image data in a block unit or a parameter, it is desirable to implement the internal memory  1515  from a semiconductor memory having a high response speed although it has a comparatively small capacity (for example, in comparison with that of the external memory  1312 ) such as, for example, an SRAM (Static Random Access Memory). 
     The codec engine  1516  performs a process relating to encoding and decoding of image data. The method of encoding and decoding with which the codec engine  1516  is compatible is arbitrary, and the number of such methods may be one or a plural number. For example, the codec engine  1516  may have a plurality of codec functions for encoding and decoding method such that encoding of image data or decoding of encoded data is performed by a selected one of the methods. 
     In the example depicted in  FIG. 48 , as functional blocks of a process relating to the codec, the codec engine  1516  includes, for example, an MPEG-2 Video  1541 , an AVC/H.264  1542 , an HEVC/H.265  1543 , an HEVC/H.265 (Scalable)  1544 , an HEVC/H.265 (Multi-view)  1545  and an MPEG-DASH  1551 . 
     The MPEG-2 Video  1541  is a functional block that encodes and decodes image data by the MPEG-2 method. The AVC/H.264  1542  is a functional block that encodes and decodes image data by the AVC method. The HEVC/H.265  1543  is a functional block that encodes and decodes image data by the HEVC method. The HEVC/H.265 (Scalable)  1544  is a functional block that performs scalable encoding or scalable encoding for image data by the HEVC method. The HEVC/H.265 (Multi-view)  1545  is a functional block that performs multi-view encoding or multi-view decoding for image data by the HEVC method. 
     The MPEG-DASH  1551  is a functional block that transmits and receives image data by the MPEG-DASH (MPEG-Dynamic Adaptive Streaming over HTTP) method. The MPEG-DASH is a technology by which streaming of a video is performed using the HTTP (HyperText Transfer Protocol), and it is one of characteristics that suitable encoded data from among a plurality of encoded data prepared in advance and having resolutions or the like different from each other is selected and transmitted in a segment unit. The MPEG-DASH  1551  performs generation of a stream in compliance with the standard, transmission control of the stream and so forth, and utilizes the MPEG-2 Video  1541  to HEVC/H.265 (Multi-view)  1545  described above in encoding and decoding of image data. 
     The memory interface  1517  is an interface for the external memory  1312 . Data supplied from the image processing engine  1514  or the codec engine  1516  is supplied to the external memory  1312  through the memory interface  1517 . Further, data read out from the external memory  1312  is supplied to the video processor  1332  (image processing engine  1514  or the codec engine  1516 ) through the memory interface  1517 . 
     The multiplexing and demultiplexing section (MUX DMUX)  1518  performs multiplexing and demultiplexing of various data relating to an image such as a bit stream of encoded data, image data, a video signal and so forth. The method for the multiplexing and demultiplexing is arbitrary. For example, upon multiplexing, the multiplexing and demultiplexing section (MUX DMUX)  1518  not only can unite a plurality of data into one data but also can add predetermined header information or the like to the data. Further, upon demultiplexing, the multiplexing and demultiplexing section (MUX DMUX)  1518  not only can divide one data into a plurality of data but also can add predetermined header information or the like to each of the divisional data. In short, the multiplexing and demultiplexing section (MUX DMUX)  1518  can convert the format of data by multiplexing or demultiplexing. For example, the multiplexing and demultiplexing section (MUX DMUX)  1518  can convert a bit stream into a transport stream that is a bit stream of a format for transfer or data of a file format for recoding (file data) by multiplexing the bit stream. Naturally, inverse conversion to the conversion is possible by demultiplexing. 
     The network interface  1519  is an interface, for example, for the broadband modem  1333 , connectivity  1321  and so forth. The video interface  1520  is an interface, for example, for the connectivity  1321 , camera  1322  and so forth. 
     Now, an example of operation of such a video processor  1332  as described above is described. For example, if a transport stream is received from an external network through the connectivity  1321 , broadband modem  1333  or the like, then the transport stream is supplied through the network interface  1519  to and demultiplexed by the multiplexing and demultiplexing section (MUX DMUX)  1518  and is decoded by the codec engine  1516 . Image data obtained by the decoding of the codec engine  1516  is subjected to a predetermined image process, for example, by the image processing engine  1514 , subjected to predetermined conversion by the display engine  1513  and supplied, for example, to the connectivity  1321  or the like through the display interface  1512  such that an image thereof is displayed on a monitor. Further, for example, the image data obtained by the decoding of the codec engine  1516  is re-encoded by the codec engine  1516 , multiplexed by the multiplexing and demultiplexing section (MUX DMUX)  1518  so as to be converted into file data, outputted, for example, to the connectivity  1321  or the like through the video interface  1520  and then recorded on various recording media. 
     Furthermore, for example, file data of encoded data encoded from image data after read out from a recording medium not depicted by the connectivity  1321  or the like is supplied through the video interface  1520  to and demultiplexed by the multiplexing and demultiplexing section (MUX DMUX)  1518  and decoded by the codec engine  1516 . Image data obtained by the decoding of the codec engine  1516  is subjected to a predetermined image process by the image processing engine  1514 , subjected to predetermined conversion by the display engine  1513 , and supplied, for example, to the connectivity  1321  or the like through the display interface  1512  such that an image is displayed on the monitor. Further, for example, the image data obtained by the decoding of the codec engine  1516  is re-encoded by the codec engine  1516 , multiplexed by the multiplexing and demultiplexing section (MUX DMUX)  1518  so as to be converted into a transport stream, supplied, for example, to the connectivity  1321 , broadband modem  1333  or the like through the network interface  1519  and then transmitted to a different apparatus not depicted. 
     It is to be noted that sending and reception of image data or other data between the processing sections in the video processor  1332  are performed, for example, utilizing the internal memory  1515  or the external memory  1312 . Further, the power management module  1313  controls power supply, for example, to the control section  1511 . 
     In the case where the present technology is applied to the video processor  1332  configured in this manner, it is sufficient if the present technology according to any embodiment described hereinabove is applied to the codec engine  1516 . In short, it is sufficient, for example, if the codec engine  1516  has the functions of the image encoding apparatus  300  or the functions of the image decoding apparatus  100  described hereinabove or both of them. This makes it possible for the video processor  1332  to achieve advantageous effects similar to those of the embodiments described hereinabove with reference to  FIGS. 1 to 40 . 
     It is to be noted that, in the codec engine  1516 , the present technology (namely, the functions of the image encoding apparatus  300 ) may be implemented by hardware such as logic circuits or may be implemented by software such as an embedded program or else may be implemented by both of them. 
     While two examples of the configuration of the video processor  1332  are exemplified above, the configuration of the video processor  1332  is arbitrary and may be any other than the two examples described above. Further, although the video processor  1332  may be configured as one semiconductor chip, it may otherwise be configured as a plurality of semiconductor chips. For example, the video processor  1332  may be formed as a three-dimensional stacked LSI in which a plurality of semiconductors are stacked. Alternatively, the video processor  1332  may be implemented by a plurality of LSIs. 
     &lt;Application Example to Apparatus&gt; 
     The video set  1300  can be incorporated into various apparatus by which image data is processed. For example, the video set  1300  can be incorporated into the television apparatus  900  ( FIG. 42 ), portable telephone set  920  ( FIG. 43 ), recording and reproduction apparatus  940  ( FIG. 44 ), imaging apparatus  960  ( FIG. 45 ) and so forth. By incorporating the video set  1300 , the apparatus can achieve advantageous effects similar to those of the embodiments described hereinabove with reference to  FIGS. 1 to 40 . 
     It is to be noted that, if even part of the components of the video set  1300  described above include the video processor  1332 , it can be carried out as a configuration to which the present technology is applied. For example, it is possible to carry out only the video processor  1332  as a video processor to which the present technology is applied. Further, it is possible to carry out the processor, video module  1311  and so forth indicated by a broken line  1341  as a processor, a module or the like to which the present technology is applied as described hereinabove. Furthermore, it is possible to combine, for example, the video module  1311 , external memory  1312 , power management module  1313  and front-end module  1314  so as to be carried out as the video unit  1361  to which the present technology is applied. In the case of any of the configurations, advantageous effects similar to those of the embodiments described hereinabove with reference to  FIGS. 1 to 40  can be achieved. 
     In short, if the video processor  1332  is included, then any configuration can be incorporated into various apparatus that process image data similarly as in the case of the video set  1300 . For example, the video processor  1332 , processor indicated by the broken line  1341 , video module  1311  or video unit  1361  can be incorporated into the television apparatus  900  ( FIG. 42 ), portable telephone set  920  ( FIG. 43 ), recording and reproduction apparatus  940  ( FIG. 44 ), imaging apparatus  960  ( FIG. 45 ) and so forth. Then, by incorporating some of the components to which the present technology is applied into an apparatus, the apparatus can achieve advantageous effects similar to those by the embodiments described hereinabove with reference to  FIGS. 1 to 40  similarly as in the case of video set  1300 . 
     Sixth Application Example: Network System 
     Also it is possible to apply the present technology to a network system including a plurality of apparatus.  FIG. 49  depicts an example of a schematic configuration of a network system to which the present technology is applied. 
     The network system  1600  depicted in  FIG. 49  is a system in which different apparatus send and receive information relating to an image (video) through a network. A cloud service  1601  of the network system  1600  is a system that provides a service relating to an image (video) to terminals connected for communication thereto such as a computer  1611 , an AV (Audio Visual) apparatus  1612 , a portable information processing terminal  1613 , an IoT (Internet of Things) device  1614  and so forth. For example, the cloud service  1601  provides a supplying service of a content of an image (video) like so-called video distribution (on-demand or live distribution) to the terminals. Further, for example, the cloud service  1601  provides a backup service of receiving and saving a content of an image (video) from the terminals. Further, for example, the cloud service  1601  provides a service of mediating the transfer of a content of an image (video) between the terminals. 
     The physical configuration of the cloud service  1601  is arbitrary. For example, the cloud service  1601  may include various servers such as a server that stores and manages videos, a server that distributes a video to the terminals, a server that acquires a video from the terminals, and a server that manages users (terminals) and accounting, or an arbitrary network such as the Internet or a LAN. 
     The computer  1611  includes an information processing apparatus such as, for example, a personal computer, a server or a work station. The AV apparatus  1612  includes an image processing apparatus such as, for example, a television receiver, a hard disk recorder, a game machine or a camera. The portable information processing terminal  1613  includes a portable information processing apparatus such as, for example, a notebook type personal computer, a tablet terminal, a portable telephone set or a smartphone. The IoT device  1614  includes an arbitrary object that performs processing relating to an image such as, for example, a machine, consumer electronics, furniture, some other article, an IC tag or a card type device. All of the terminals mentioned have a communication function and can connect to (establish a session with) the cloud service  1601  to perform sending and reception of information to and from (namely, to perform communication with) the cloud service  1601 . Further, each terminal can perform communication also with the other terminals. Communication between the terminals may be performed through the cloud service  1601  or may be performed without the intervention of the cloud service  1601 . 
     The present technology may be applied to such a network system  1600  as described above such that, when data of an image (video) is to be transferred between the terminals or between the terminals and the cloud service  1601 , the image data is encoded and decoded in such a manner as described hereinabove in connection the embodiments. In short, the terminals (computer  1611  to IoT device  1614 ) and the cloud service  1601  may individually have the functions of the image encoding apparatus  300  or the image decoding apparatus  100  described hereinabove. This makes it possible for the terminals (computer  1611  to IoT device  1614 ) that send and receive image data and the cloud service  1601  to achieve advantageous effects similar to those by the embodiments described hereinabove with reference to  FIGS. 1 to 40 . 
     &lt;Others&gt; 
     It is to be noted that various kinds of information relating to encoded data (bit stream) may be multiplexed into and transmitted or recorded together with the encoded data or may be transmitted or recorded as separate data associated with the encoded data without being multiplexed with the encoded data. Here, the terminal “associate” signifies to make it possible, for example, when one data is to be processed, to utilize (link with) the other data. In short, data associated with each other may be united into one data or may remain individual data. For example, information associated with encoded data (image) may be transmitted on a transmission line separate from that for the encoded data (image). Further, for example, information associated with encoded data may be recorded on a recording medium different from that for the encoded data (image) (or into a recording area of the same recording medium). It is to be noted that this “association” may be not of entire data but of part of data. For example, an image and information corresponding to the image may be associated with each other in an arbitrary unit such as a plurality of frames, one frame or part in a frame. 
     Further, as described hereinabove, such terms in the present specification as “synthesize,” “multiplex,” “add,” “unite,” “include,” “store,” “put in,” “plug in” and “insert” signify to combine multiple things to one such as, for example, to combine encoded data and metadata into one data and each signifies one method of the “association” described above. 
     Further, the embodiment of the present technology is not limited to the embodiments described hereinabove, and various alterations are possible without departing from the subject matter of the present disclosure. 
     For example, in the present specification, the term system signifies a set of plural components (apparatus, modules (parts) and so forth) and does not matter whether or not all components are placed in a same housing. Accordingly, both of a plurality of apparatus that are accommodated in separate housings and are connected to each other by a network and one apparatus in which a plurality of modules are accommodated in one housing are systems. 
     Further, for example, a constitution described as one apparatus (or one processing section) may be divided into and configured as a plurality of apparatus (or processing sections). Conversely, constitutions described as a plurality of apparatus (or processing sections) in the foregoing description may be collected such that they are configured as one apparatus (or one processing section). Further, a constitution other than those may naturally be added to the configuration of each apparatus (or each processing section). Furthermore, if a constitution or operation as an entire system is substantially same, then part of constitutions of a certain apparatus (or a certain processing section) may be included in constitutions of a different apparatus (or a difference processing section). 
     Further, for example, the present technology can assume a configuration for cloud computing in which one function is shared and processed in cooperation by a plurality of apparatus through a network. 
     Further, for example, the program described hereinabove can be executed by an arbitrary apparatus. In this case, it is sufficient if the apparatus is configured such that it has necessary functions (functional blocks and so forth) and can acquire necessary information. 
     Further, for example, the steps described in connection with the flow charts described hereinabove can be executed by one apparatus and further can be shared and executed by a plurality of apparatus. Furthermore, in the case where a plurality of processes are included in one step, the plurality of processes included in the one step can be executed by one apparatus and also can be shared and executed by a plurality of apparatus. 
     It is to be noted that the program to be executed by the computer may be of the type by which the processes at steps by which the program is described are executed in a time series in the order as described in the present specification or of the type by which the processes are executed in parallel or executed individually at necessary timings such as when the process is called. Furthermore, the processes at the steps by which the program is executed may be executed in parallel to processes of a different program or may be executed in combination with processes of a different apparatus. 
     It is to be noted that the plurality of present technologies described in the present specification can individually be carried out solely and independently of each other unless inconsistency occurs. Naturally, also it is possible to carry out an arbitrary plurality of present technologies in combination. For example, also it is possible to carry out the present technology described in the description of any embodiment in combination with the present technology described in the description of a different embodiment. Also it is possible to carry out an arbitrary one of the present technologies described hereinabove in combination with a different technology that is not described hereinabove. 
     It is to be noted that the present technology can assume also such a configuration as described below. 
     (1) 
     An image processing apparatus, including: 
     a decoding section configured to decode encoded data; 
     an inverse primary vertical transform controlling section configured to control, based on a value of a transform skip identifier obtained by the decoding of the encoded data by the decoding section, execution of an inverse primary vertical transform that is an inverse primary transform in a vertical direction for transform coefficient data transformed from image data; and 
     an inverse primary horizontal transform controlling section configured to control, based on the value of the transform skip identifier, execution of an inverse primary horizontal transform that is an inverse primary transform in a horizontal direction for the coefficient data transformed from the image data. 
     (2) 
     The image processing apparatus according to (1), in which 
     the inverse primary vertical transform controlling section controls the execution of the inverse primary vertical transform such that,
         where the transform skip identifier indicates that a one-dimensional transform in the vertical direction is not to be skipped, the inverse primary vertical transform for the transform coefficient data is executed, but   where the transform skip identifier indicates that a one-dimensional transform in the vertical direction is to be skipped, the inverse primary vertical transform for the transform coefficient data is omitted.
 
(3)
       

     The image processing apparatus according to (1) or (2), in which 
     the inverse primary horizontal transform controlling section controls the execution of the inverse primary horizontal transform such that,
         where the transform skip identifier indicates that a one-dimensional transform in the horizontal direction is not to be skipped, the inverse primary horizontal transform for the transform coefficient data is executed, but   where the transform skip identifier indicates that a one-dimensional transform in the horizontal direction is to be skipped, the inverse primary horizontal transform for the transform coefficient data is omitted.
 
(4)
       

     The image processing apparatus according to any one of (1) to (3), further including: 
     a selection section configured to select an orthogonal transform that is to be applied to the inverse primary vertical transform and the inverse primary horizontal transform. 
     (5) 
     The image processing apparatus according to (4), in which 
     the selection section
         selects an orthogonal transform to be applied as the inverse primary vertical transform based on a vertical transform set identifier and a primary vertical transform designation flag obtained by the decoding of the encoded data by the decoding section, and   selects an orthogonal transform to be applied as the inverse primary horizontal transform based on a horizontal transform set identifier and a primary horizontal transform designation flag obtained by the decoding of the encoded data by the decoding section.
 
(6)
       

     The image processing apparatus according to (5), in which 
     the decoding section derives the primary vertical transform designation flag and the primary horizontal transform designation flag from a primary transform identifier in response to the value of the transform skip identifier. 
     (7) 
     The image processing apparatus according to (6), in which 
     the decoding section
         derives, where the transform skip identifier indicates that a two-dimensional transform is not to be skipped, the primary vertical transform designation flag and the primary horizontal transform designation flag by processing the primary transform identifier as a 2-bit bin string, and   derives, where the transform skip identifier indicates that a one-dimensional transform in the vertical direction or the horizontal direction is not to be skipped, the primary vertical transform designation flag and the primary horizontal transform designation flag by processing the primary transform identifier as a 1-bit bin string.
 
(8)
       

     The image processing apparatus according to any one of (5) to (7), in which 
     the decoding section decodes the primary vertical transform designation flag and the primary horizontal transform designation flag included in the encoded data. 
     (9) 
     The image processing apparatus according to any one of (1) to (8), in which, 
     where the transform skip identifier indicates that a one-dimensional transform in the vertical direction or the horizontal direction or a two-dimensional transform is not to be skipped, the decoding section omits decoding of a secondary transform identifier and sets the secondary transform identifier to a value that indicates that a secondary transform is not to be performed. 
     (10) 
     An image processing method, including: 
     decoding encoded data; 
     controlling, based on a value of a transform skip identifier obtained by the decoding of the encoded data, execution of an inverse primary vertical transform that is an inverse primary transform in a vertical direction for transform coefficient data transformed from image data; and 
     controlling, based on the value of the transform skip identifier, execution of an inverse primary horizontal transform that is an inverse primary transform in a horizontal direction for the coefficient data transformed from the image data. 
     (11) 
     An image processing apparatus, including: 
     a primary horizontal transform controlling section configured to control execution of a primary horizontal transform that is a primary transform in a horizontal direction for residual data between an image and a prediction image based on a value of a transform skip identifier; 
     a primary vertical transform controlling section configured to control, based on a value of the transform skip identifier, execution of a primary vertical transform that is a primary transform in a vertical direction for the residual data between the image and the prediction image; and 
     an encoding section configured to encode the transform skip identifier. 
     (12) 
     The image processing apparatus according to (11), in which 
     the primary horizontal transform controlling section controls the execution of the primary horizontal transform such that,
         where the transform skip identifier indicates that a one-dimensional transform in the horizontal direction is not to be skipped, the primary horizontal transform for the residual data is executed, but   where the transform skip identifier indicates that a one-dimensional transform in the horizontal direction is to be skipped, the primary horizontal transform for the residual data is omitted.
 
(13)
       

     The image processing apparatus according to (11) or (12), in which 
     the primary vertical transform controlling section controls the execution of the primary horizontal transform such that,
         where the transform skip identifier indicates that a one-dimensional transform in the vertical direction is not to be skipped, the primary vertical transform for the residual data is executed, but   where the transform skip identifier indicates that a one-dimensional transform in the vertical direction is to be skipped, the primary vertical transform for the residual data is omitted.
 
(14)
       

     The image processing apparatus according to any one of (11) to (13), further including: 
     a selection section configured to select an orthogonal transform that is to be applied to the primary horizontal transform and the inverse primary vertical transform. 
     (15) 
     The image processing apparatus according to (14), in which 
     the selection section
         selects an orthogonal transform to be applied as the primary horizontal transform based on a horizontal transform set identifier and a primary horizontal transform designation flag, and   selects an orthogonal transform to be applied as the primary vertical transform based on a vertical transform set identifier and a primary vertical transform designation flag.
 
(16)
       

     The image processing apparatus according to (15), in which 
     the encoding section derives a primary transform identifier from the primary horizontal transform designation flag and the primary vertical transform designation flag in response to the value of the transform skip identifier. 
     (17) 
     The image processing apparatus according to (16), in which 
     the encoding section
         derives, where the transform skip identifier indicates that a two-dimensional transform is not to be skipped, the primary transform identifier of a 2-bit bin string using the primary horizontal transform designation flag and the primary vertical transform designation flag, and   derives, where the transform skip identifier indicates that a one-dimensional transform in the vertical direction or the horizontal direction is not to be skipped, the primary transform identifier of a 1-bit bin string using the primary horizontal transform designation flag or the primary vertical transform designation flag.
 
(18)
       

     The image processing apparatus according to any one of (15) to (17), in which 
     the encoding section encodes the primary horizontal transform designation flag and the primary vertical transform designation flag. 
     (19) 
     The image processing apparatus according to any one of (11) to (18), in which, 
     where the transform skip identifier indicates that a one-dimensional transform in the vertical direction or the horizontal direction or a two-dimensional transform is not to be skipped, the encoding section omits encoding of a secondary transform identifier. 
     (20) 
     An image processing method, including: 
     controlling execution of a primary horizontal transform that is a primary transform in a horizontal direction for residual data between an image and a prediction image based on a value of a transform skip identifier; 
     controlling, based on a value of the transform skip identifier, execution of a primary vertical transform that is a primary transform in a vertical direction for the residual data between the image and the prediction image; and 
     encoding the transform skip identifier. 
     REFERENCE SIGNS LIST 
     
         
         
           
               100  Image decoding apparatus,  111  Decoding section,  112  Dequantization section,  113  Inverse transform section,  114  Arithmetic operation section,  115  Frame memory,  116  Prediction section,  121  Switch,  122  Inverse secondary transform section,  123  Inverse primary transform section,  131  Inverse primary transform selection section,  132  Switch,  133  Inverse primary vertical transform section,  134  Switch,  135  Inverse primary horizontal transform section,  151  Transform skip validity flag decoding section,  152  Maximum transform skip block size decoding section,  153  Transform quantization bypass flag decoding section,  154  Transform skip identifier decoding section,  161  Primary transform validity flag decoding section,  162  Adaptive primary transform flag decoding section,  163  Primary transform identifier decoding section,  171  Secondary transform validity flag decoding section,  172  Secondary transform identifier decoding section,  181  Primary horizontal transform designation flag decoding section,  182  Primary vertical transform designation flag decoding section,  191  and  192  Scaling section,  300  Image encoding apparatus,  311  Control section,  312  Arithmetic operation section,  313  Transform section,  314  Quantization section,  315  Encoding section,  316  Dequantization section,  317  Inverse transform section,  318  Arithmetic operation section,  319  Frame memory,  320  Prediction section,  331  Switch,  332  primary transform section,  333  Secondary transform section,  341  Primary transform selection section,  342  Switch,  343  Primary horizontal transform section,  344  Switch,  345  Primary vertical transform section,  361  Transform skip validity flag encoding section,  362  Maximum transform skip block size encoding section,  363  Transform quantization bypass flag encoding section,  364  Transform skip identifier encoding section,  371  Primary transform validity flag encoding section,  372  Adaptive primary transform flag encoding section,  373  Primary transform identifier encoding section,  381  Secondary transform validity flag encoding section,  382  Secondary transform identifier encoding section,  391  Primary horizontal transform designation flag encoding section,  392  Primary vertical transform designation flag encoding section,  401  and  402  Scaling section