Patent Publication Number: US-9424808-B2

Title: Image cropping manipulation method and portable electronic device

Description:
BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention relates to an image cropping manipulation method and a portable electronic device thereof, and more particularly, a cropping manipulation method for updating content of a cropped image according to user inputs. 
     2. Description of the Prior Art 
     Image cropping is one of the image editing tools often used by user. Conventionally, image cropping applications provide a cropping frame of certain shape, such as rectangular, circle or heart shape, etc. and user may move the cropping frame to desired position within the image and adjust the size of the cropping frame. The image cropping application would crop a sub-image according to the cropping frame and store the sub-image for other use. For example, the sub-image may be stored as a separate image, integrated with other images or image templates. Though the position and the size of the cropping frame can be designated by user, the respective shape and image distortion cannot be solved by image cropping application. For example, if an image comprising an object is taken in a non-front view angle, the shape of the object appears in the digital image would be distorted. For example, an object in rectangular shape might become in trapezoid shape, another object in round shape might become in elliptical. When cropping the object from the digital image, the distortion caused by the inclination will also be reflected in the cropped sub-image. This would cause inconvenient or uncomfortable viewing to the user, particularly in the case that the sub-image comprises text or other useful information. Fixing the distortion caused by viewing angle would be beneficial and improve image quality. 
     To solve above issues, the present invention provides a novel approach to manipulate image viewing angle and adjust distortion caused thereof. 
     SUMMARY OF THE INVENTION 
     In an embodiment of the invention, a portable electronic device for performing image cropping manipulation is disclosed. The portable electronic device comprises: a touch screen configured to display a sub-image and to provide a user interface for receiving at least one user input, the sub-image is cropped from a digital image; and an image processing unit configured to perform an operation corresponding to the user input. The image processing unit comprises: a space mapping module, configured to map the sub-image to an image space and map the user input to the operation in the image space; and an image cropping module, configured to perform the operation and update the sub-image according to the user input. 
     In another embodiment of the invention, an image cropping manipulation method for use in a portable electronic device is disclosed. The portable electronic device comprises a touch screen and an image processing unit, and the method comprises: displaying a sub-image on the touch screen, the sub-image is cropped from a digital image; receiving a user input from the touch screen; in response to the user input, map the user input to an operation on the digital image; and performing the operation to update the sub-image. 
     These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  illustrates an image cropping manipulation for cropping a sub-region from a digital image according to one embodiment of the present invention. 
         FIG. 2  illustrates a portable electronic device for implementing the image cropping manipulation method according to one embodiment of the present invention. 
         FIGS. 3-11  illustrate embodiments of image cropping manipulation of the invention in image space and cropping space and mapping mechanisms of the two spaces. 
         FIG. 12-13  illustrate image cropping manipulation methods according to embodiments of the present invention. 
     
    
    
     DETAILED DESCRIPTION 
     To solve the problem addressed above, the present invention discloses an image cropping manipulation method and a portable electronic device implementing the image cropping manipulation method. In the disclosed image cropping manipulation method, a plurality of control points corresponding to a sub-region within a digital image are defined and adjusted in an image space, and a sub-image corresponding to the sub-region is determined and cropped from the digital image according to the control points. 
     Please refer to  FIG. 1 , which schematically illustrates an embodiment of image cropping manipulation. The image cropping manipulation may be performed on a mobile electronic device, such as a smart phone or tablet, and the mobile electronic device may comprise a touch screen and a camera. As shown in  FIG. 1 , a digital image I 1  is retrieved and provided to a user. The digital image I 1  can be captured by the camera and displayed on the touch screen. The digital image I 1  can also be accessed from an internal storage unit, external storage device, or wirelessly from another data source, such as social service provider. The mobile electronic device may provide a user interface on the touch screen for user to perform operations on digital image I 1 . In embodiments of the invention, the user interface provides an image cropping function for user to crop a sub-image C 1  from the digital image I 1 . The resulting sub-image C 1  is displayed and can be further stored as a single image. Prior to storing the sub-image C 1 , user may send inputs to modify the sub-image C 1  via the touch screen. For example, user may edit content of the sub-image C 1 . As shown in the embodiment of  FIG. 1 , the digital image I 1  comprises an object M, which is a painting board. Each of the corners B 1 ˜B 4  of the sub-image corresponds to one of the control points A 1 ˜A 4  in the digital image I 1  respectively. To generate the sub-image C 1 , a sub-region R 1  defined by the control points A 1 ˜A 4  is cropped from the digital image I 1 . The sub-image C 1  may be properly scaled to desired resolution, such as full resolution. When user manipulates on the sub-image C 1 , position of corresponding control points A 1 ˜A 4  would be updated accordingly, and consequently the content of the sub-image C 1  is updated. 
       FIG. 2  illustrates a portable electronic device  10  for performing image cropping manipulation according to an embodiment of the invention. The portable electronic device  10  comprises, among other components, a touch screen  100 , an image processing unit  200 , a camera module  300  and a storage unit  400 . The touch screen  100  is configured to display digital images and provide a user interface for receiving user inputs. The touch screen  100  further comprises touch panel  110  for receiving the user inputs and a display module  120  for displaying digital images and user interfaces. The image processing unit  200  is configured to access digital images and to perform various processing on the digital images. Digital images can be accessed from various image sources, such as the camera module  300  and/or the storage unit  400 . Furthermore, the image processing unit  200  is configured to receive user inputs from the touch screen  100  and perform operations corresponding to the user inputs on the digital images. The processed digital images may be provided back to the touch screen  100  for display and/or be stored in the storage unit  400 . 
     The image processing unit  200  further comprises at least a space mapping module  210  and an image cropping module  220 . The space mapping module  210  is configured to define an image space and a cropping space, and convert user inputs received in one space to another. The image cropping module  220  is configured to crop a sub-region of a digital image according to the user input and provide a sub-image corresponding to the sub-region. The image cropping module  220  is further configured to adjust pixel values and/or positions in the sub-image according to the user input. 
     Described in more details with reference to  FIG. 1 , the touch screen  100  displays the digital image I 1  and receives a user input for cropping a sub-region R 1  from the digital image I 1 . Upon receiving the user input, the touch screen  100  may transmit the user input to the image processing unit  200 . The space mapping module  210  may determine the position of the sub-region R 1  in the digital image I 1 , and the image cropping module  210  crops and provides a corresponding sub-image C 1 . The sub-image C 1  is then transmitted to the touch screen  100  for displaying to the user. As shown in  FIG. 1 , the space mapping module  210  defines an image space corresponding to the digital image I 1  and a cropping space corresponding to the sub-image C 1 . The sub-region R 1  is defined by four control points A 1 ˜A 4  in the image space. The image cropping module  220  crops the sub-region R 1  from the digital image I 1  according to the control points A 1 ˜A 4 . Please note that the touch screen  100  may display a preview version of the digital image I 1 , which is of lower resolution. And the image cropping module  220  may perform image cropping on the digital image I 1  in full resolution. The image cropping module  220  also performs other modifications such as scaling and/or morphing on the cropped sub-region M to generate the sub-image C 1 . Specifically, the image cropping module  220  may transform the sub-image C 1  from a non-rectangular shape in the image space into rectangular shape in the cropping space. 
     Please refer to  FIGS. 3 and 4 , which illustrate an operation of image cropping manipulation according to an embodiment of the invention. As shown in the left of  FIG. 3  and with reference to  FIG. 2 , the image copping module  220  generates a sub-image C 2  cropped from a digital image I 2  and provides it for displaying on the touch screen  100  to the user. The digital image I 2  is assumed to be in an image space and the sub-image C 2  is assumed to be in a cropping space. The control points A 1 ˜A 4  define the sub-region R 2  corresponding to the sub-image C 2  in the image space. User inputs received in the cropping space can be mapped to corresponding operations in the image space by the space mapping module  210 . In this embodiment, the sub-image C 2  initially does not fully cover the object N in the digital image I 2 . User may send user inputs via the touch screen  100  to update content of the sub-image C 2 . Since the most left part of object N is not shown, user may drag his/her finger from left to right on the touch screen  100  until the sub-image C 2  is fully covered by the object N, as shown in the right of  FIG. 3 . To update the content of the sub-image C 2 , the space mapping module  210  maps the drag user input in the cropping space to operations on the control points A 1 ˜A 4  in the image space. The control points A 1 ˜A 4  are moved from right to left in relative opposite direction to the user input. In this embodiment of the invention, the control points A 1 ˜A 4  are moved synchronously. However, in other embodiments of the invention, each of the control points A 1 ˜A 4  may be adjusted independently. 
       FIG. 5  illustrates a space mapping mechanism according to an embodiment of the invention. The space mapping mechanism may be performed by the space mapping module  210 . In the cropping space where the sub-image C 2  lies in, a plurality of control regions T 1 ˜T 5  are defined. At least a portion of the control regions, in this embodiment T 1 ˜T 4 , correspond to the control points A 1 ˜A 4  in the image space respectively. For a given control region, user inputs received in that region are mapped to operations of corresponding control points. For example, user inputs received in control region T 1  are mapped to operations of the control point A 1 . By manipulation of user input in the control region T 1 , user may change position of the control point A 1  in the image space and thus change content of the sub-image C 2  nearby control region T 1 . For the center control region T 5 , user inputs received in this region would be mapped to all of the control points A 1 ˜A 4 , causing them to move positions synchronously. 
     Please note that, in the image space the control points A 1 ˜A 4  may not form a rectangular or square as the sub-image C 2  does. As a result, the moving direction of the user inputs in each control region would be remapped to proper directions in the image space. The image space may be regarded as a vector space formed by the control points A 1 ˜A 4 . The control point A 1  may comprise four principal vectors with respect to other control pints A 2 ˜A 4 . Vector V 11  points from control point A 1  to A 3 , vector V 12  points from A 2  to A 1 , vector v 13  points from A 3  to A 1  and vector V 14  points from A 1  to A 2 . The drag user input in the control region T 1  also forms an input vector, and the input vector is mapped to an output vector according to the vectors V 11 ˜V 14 . Similarly, control point A 2  has principal vectors V 21 ˜V 24 , and user inputs in the control region T 2  are mapped to output vectors according to principal vectors V 21 ˜V 24 . Control point A 3  has principal vectors V 31 ˜V 34 , and user inputs in the control region T 3  are mapped to output vectors according to principal vectors V 31 ˜V 34 . And control point A 4  has principal vectors V 41 ˜V 44 , and user inputs in the control region T 4  are mapped to output vectors according to principal vectors V 41 ˜V 44 . For user inputs in the control region T 5 , if the user input is in horizontal direction from left to right, as input vector D 1 , control points A 1 ˜A 4  should be horizontally from left to right accordingly. For example, control points A 1  and A 3  would move along principal vectors V 11  and V 31  accordingly. Similarly, if the user input is in vertical direction in the control region T 5 , control points A 1  and A 2  would move together along the vector formed in-between and control points A 3  and A 4  would move together. 
     To better illustrate the movement of the control points in response to user inputs, several embodiments will be described in below. However, please note that the invention is not limited to these embodiments. Modifications or variations may be made without departing from the spirit of the invention. Please refer to  FIGS. 6 ˜ 8 , which illustrate an example of user input received in a specific control region. As shown in left of  FIG. 6 , a sub-image C 3  is provided on the touch screen  100  to the user. It can be observed that there exists large distortion at the top-left region of the sub-image C 3 . In this case, user might wish to rectify the distortion and bring in more contents. To achieve this purpose, user may send a drag user input from the top-left corner of the cropping image C 3  to an inner position. The drag user input in the cropping space is shown in the left of  FIG. 7  and converted as input vector D 14 . As shown in  FIG. 7 , the input vector D 14  occurs in the control region T 1 , and maps to an output vector V 123  with respect to control point A 1  in the image space. Please note that the output vector V 123  is in relative opposite direction to the input vector D 14 . That is, when user drags from an outer position to inner position on the touch screen  100 , the attempt is to move the dragged content further inside and reveals uncovered outer contents. To include the previously uncovered contents, the control point has to extend further outside. As a result, the input vector and the output vector point to opposite directions. 
     Further explained, the input vector D 14  is determined by the trace of the user input (for example start point and end point), and can be defined by the combination of the principal vectors D 1  and D 4  in the cropping space. As can be understood, the cropping space is in rectangular shape with right angles. Thus, each vector in the cropping space can be composed as a combination of two principal vectors, the two vectors form a right angle in between, for example D 1  and D 4  in  FIG. 7 . However, in the image space the control points X 1 ˜X 4  may not form a rectangle or a square, the angles between the principal vectors may not be right angles. Thus vector mapping should be performed to convert the input vector in the cropping space (for example D 14 ) to corresponding output vector in the image space (for example V 123 ). The mapping can be done by determining the ratio of a first angle between D 14  and D 1  (or D 4 ) and a second angle between V 123  and V 13  (or V 12 ). And the moving distance of V 123  can be mapped from the relative moving distance of D 14 . 
     In an embodiment of the invention, the control point X 1  and the displayed cropping image C 3  can be updated with the drag user input in real-time. That is, when user places his/her finger on the touch screen, the sub-image C 3  is updated along with finger movement. The user may stop dragging upon a desired cropping image C 3  is disclosed. In this case, the control point X 1  is updated with the finger movement as well so that the sub-image C 3  can be updated. Please note that the relative vectors V 11 ˜V 14  corresponding to vectors D 1 ˜D 4  may also be updated in response to the update of control point X 1 . In other words, when the control point X 1  moves its position, the axes form by the control points X 1 ˜X 4  would change accordingly. As a result, the pointing directions of the principal vectors V 11 ˜V 14  relative to control point X 1  would change, and also does the output vector V 123  in consequent. 
       FIG. 8  depicts image cropping in the image space. The sub-region R 3  corresponding to the control points X 1 ˜X 4  is cropped by the image cropping module  220  and displayed as the sub-image C 3  of  FIG. 6 . The control point X 1  initially lies inside the object M desired to be cropped. Upon receiving the drag user input from the user, the control point X 1  is moved to desired position, which is the top-left corner of object M, as shown in the right of  FIG. 8 . Please note that even though the sub-region R 3  appears in the digital image I 3  in a non-rectangular shape, it is mapped into the cropping space in rectangle. Therefore, distortions appears at edge, corner or other portions in sub-region R 3  can be rectified in the sub-image C 3 , as shown in comparison of  FIG. 6  and  FIG. 8 . Pixel values and positions may be adjusted according to the movement of control points, and thus the scale and aspect ratio of the sub-image C 3  is adjusted. In the condition that an object within an image is not taken in front view, by cropping and adjusting control points of a sub-region according to the invention, the object can be presented in front view for better view. 
       FIGS. 9, 10 and 11  illustrate another embodiment of image cropping of the invention. In this embodiment, the user interface is capable to receive multiple drag user inputs corresponding to multiple control points simultaneously. First please refer to FIG.  9 , a sub-image C 4  is provided and displayed on a user interface for of the touch screen  100 . The sub-image C 4  corresponds to a sub-region R 4  in the digital image I 4  and is defined by control points X 1 ˜X 4 . As shown in the left of  FIG. 10 , the top-left and bottom-right portions of the digital image I 4  are not covered by the sub-region R 4 . To reveal those uncovered portions, the user may use two fingers to drag from the top-left and bottom-right corners to the center simultaneously ( FIG. 9 ). Corresponding control points X 1  and X 4  would move outward in the top-left and bottom-right directions respectively ( FIG. 10 ). As described above, the two user inputs are located in the control region T 1  and T 4  respectively and are converted into two input vectors D 14  and D 67 . The input vectors D 14  and D 67  are mapped into the image space as output vectors V 123  and V 414 , but the input vectors and the output vectors are of relative opposite directions. The mapping of input vector D 14  and the output vector V 123  is similar to the embodiment of  FIG. 7 . The mapping of D 67  and V 414  can be done by determining the ratio of a first angle between D 67  and D 6  (or D 7 ) and a second angle between V 414  and V 44  (or V 41 ). And the moving distance of V 414  can be mapped from the relative moving distance of D 67 . 
     Please note that the user input may be a single input corresponding to one control point or multiple inputs corresponding to multiple control points. In the latter case, the multiple inputs may correspond to any combination of the control points, such as X 1  and X 4  in  FIGS. 10 and 11 , X 1  and X 2 , or even X 1 , X 2  and X 3 . 
     Please refer to  FIG. 12 .  FIG. 12  illustrates an image cropping manipulation method according to an embodiment of the invention. The image cropping manipulation method may be implemented in a portable electronic device, such as a smart phone, tablet, and/or others. The portable electronic device may comprise touch screen for displaying images and providing a user interface to receive user inputs. The portable electronic device may also comprise an image processing unit for performing various image processing. Starting from step S 1210 , a digital image is displayed on the touch screen. The digital image can be captured by a camera module of the portable electronic device or be accessed from internal or external storage unit. Next the touch screen receives a first user input corresponding to a sub-region of the digital image in step S 1220 . As mentioned, the touch screen may provide user interface for receiving user inputs corresponding to various operations. In the embodiment of the invention, the user interface may comprise image edit functions, such as rotate, crop, effect, etc. User may select a desired function via the user interface, for example crop in this embodiment. The user input may comprise information of a sub-region to be cropped. In step S 1230 , the image processing unit of the portable electronic device crops the sub region from the digital image and provides a corresponding sub-image on the touch screen. Please note that the digital image displayed on the touch screen may be of preview resolution, which is lower than original resolution. The sub region may be cropped from the digital image of full resolution, and be processed prior to generate the sub-image. 
     While viewing the sub-image, user may wish to apply further operations on the sub-image. The touch screen may also provide user interface for receiving user inputs corresponding to operations on the sub-image. For example, the cropped sub-image may not be satisfying and the user wants to modify the scope of the sub-image. In step S 1240 , the touch screen receives a second user input for editing the sub-image. The user input may be a single-touch or multi-touch gesture, and comprise information of position(s) user desires to modify. For example, user may apply a drag input moving from left to right to indicate that he/she wants to crop a sub-region further left in the digital image to the currently displayed sub-image. In another example, user may drag from one or more corners of the sub-image to extend the sub-region broader in the digital image. Please note that user may directly send user inputs on the sub-image to modify its content without returning to the digital image for selecting a new sub-region to be cropped. In step S 1250 , the user input is mapped to an operation on the digital image. In embodiments of the invention, the image processing unit defines an image space corresponding to the digital image and a cropping space corresponding to the sub-image. The user input received from the touch screen in first determined in the cropping space and converted to an operation in the image space. That is to say, the actual operation takes place on the digital image, however the user input is received on the sub-image. A plurality of control regions corresponding to the sub-image are determined in cropping space, and a plurality of control points corresponding to the sub-region are determined in the image space. User inputs fall in one or more of the control regions can be mapped to adjustment of one or more control points. The image processing unit may convert the user input to an input vector in corresponding control region, and map the input vector to an output vector with respect to corresponding control point in the image space. In step S 1260 , the image processing unit performs corresponding operation to update the sub-image. The control point is moved according to the output vector, leading to the adjustment of the sub-region. By adjusting the positions of the control points, the sub-region is adjusted accordingly and thus the sub-image is updated. Please note that the input vector and the output vector are of relative opposite direction. For example, if the input vector points from left to right (i.e. user input is a drag from left to right), the output vector points from right to left (i.e. control point moves from right to left). 
       FIG. 13  illustrates another embodiment of image cropping manipulation method of the invention. Similar to the embodiment of  FIG. 12 , this embodiment may be implemented in a portable electronic device comprising a touch screen and an image processing unit. In step S 1310 , a sub-image is displayed on the touch screen. The sub-image corresponds to a sub-region cropped from a digital image. In step S 1320 , a user input is received from the touch screen. The touch screen may provide user interface for receiving user inputs corresponding to various operations to the sub-image. In step S 1330 , the image processing unit defines an image space corresponding to the digital image and a cropping space corresponding to the sub-image. Please note that step S 1330  may be performed prior to step S 1320 . In the image space, a sub-region corresponding to the sub-image is defined by a plurality of control points. In the cropping space, a plurality of control regions is defined. Each of the control regions corresponds to one or more of the control points. In step S 1340 , the image processing unit maps the user input from the cropping space to the image space. The image processing unit determines one or more control regions in which the user input is received. If the user input is a single-touch gesture, a corresponding control region is determined. If the user input is a multi-touch gesture, control regions corresponding to the multi-touch are determined. The movement of the user input is converted to an input vector in the corresponding control region. And the input vector is then mapped to an output vector of the corresponding control point. In step S 1350 , the image processing unit adjusts the sub-region in the image space according to the user input. The image processing unit adjusts the control point according to the output vector, which is mapped and converted from the user input. As described earlier, the sub-region is defined by the control points. By adjusting positions of the control points, the sub-region is adjusted accordingly and again cropped from the digital image. Then in step S 1360 , the sub-image is updated accordingly on the touch screen. Please note that the update of the sub-image may be performed synchronously with the receiving of the user input. That is to say, the sub-image is updated along with the movement of the user input with minimum delay or without delay. Also, the user input is directly applied during the display of the sub-image. Although operations of the user input are performed on the digital image in image space, the user interface stays in the cropping space of the sub-image. Last in step S 1370 , the updated sub-image may be stored in storage unit of the portable electronic device or an external storage device. For example, user may store the sub-image in the memory of the portable electronic device or upload the sub-image to a social media server. 
     Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.