A method of automatically orienting symmetric and asymmetric produce items, such as apples for example, is provided. Individual items of produce are manipulated by a programmable manipulator within the view of one or more depth imaging cameras. Digital three dimensional characterizations of the surface of the produce items are generated by the depth imaging camera or cameras and are utilized by a computer connected to the depth imaging camera or cameras to locate the stem and blossom of each produce item. Asymmetric produce items, such as apples with dropped shoulders as well as symmetric produce items can be properly oriented and processed automatically.

BACKGROUND

The present invention pertains in general to the automatic orienting of produce items, so that those items can subsequently be processed automatically by peelers, coring devices and slicers, for example. The following description and drawings will show and describe the invention primarily as utilized in automatically orienting apples. However, it is to be understood that the invention can be applied to other produce items such as peaches, pears and apricots, for example.

In the case of automatically orienting apples, the prior art has typically concentrated on locating either the stem indent, in which the stem is located, or the blossom indent, in which the blossom is located, of the apple, and then assumes that the body of the apple is symmetrical along an axis from the blossom indent to the stem indent, i.e. that the stem and blossom are vertically aligned. This assumption is valid for some apples, but not all. Many apples (5% in some varieties) are not symmetrical along an axis from the blossom to the stem. Furthermore, changes in apple varieties, cultivation practices and climate have resulted in an increase in the percentage of “dropped shoulder” apples, which are asymmetrical. When such an asymmetric apple is transferred either blossom indent or stem indent downwardly to a common automatic vertically mounted coring tube, the coring tube enters the apple at a point away from the uppermost indent and does not remove the entire core of the apple, and undesirable seeds and other core materials are not removed. That asymmetric apple cannot be used, and to make matters worse, it must be separated and disposed of. The result is the loss of the apple, the effort and expense required to detect and separate that apple, and the added expense of disposing of the remains of that apple. If an asymmetric apple is not detected and separated the undesirable seeds and core material may reduce the quality and/or usability of a large number of the processed apples

The present invention, for the first time known to the applicants, provides a system for automatically locating the stem and blossom of an apple, regardless of the shape of the apple. Once the stem and blossom are automatically located, the apple is automatically manipulated to align the apple so that the automatic coring of the apple will successfully remove all of the unwanted seeds and other undesirable core material from the apple.

The present invention reduces the waste otherwise caused by using typical prior art automatic orienting apparatus, eliminates the effort and expense of locating and separating asymmetric apples and eliminates the expense of disposing of the remains of apples not successfully cored or otherwise not successfully processed.

A typical prior art apple orientor is shown in Tichy U.S. Pat. No. 4,746,001. Apples are conveyed singularly into a receptacle in which apples are rotated by wheels below the apple until either the blossom indent or stem indent loses contact with the wheels and the apple comes to rest at the base of receptacle. The apple is then assumed to be oriented and is transferred to a vertical coring machine. However, if the apple is asymmetrical about an axis extending from its blossom to its stem, it will not be successfully cored, resulting in the losses and added expense noted above.

The Ross et al U.S. Pat. No. 5,544,731 and Amstad U.S. Pat. No. 4,169,528 teach apple orientors which either agitate or rotate apples until either end of the apple comes to rest at the base of a receptacle. These devices also transfer the apple to a vertically actuated coring tube, assuming the upper end of the apple to be vertically aligned with the lower end of the apple. As noted above, the coring of asymmetric apples with these orientors is not successful.

The Throop et al U.S. Pat. No. 5,855,270 teaches an apple or other produce orienting device using a pair of opposed rollers on horizontal axes to cause the stem and blossom to be oriented on a horizontal axis. This system also does not properly orient asymmetric apples.

The prior art includes pear orienting machines described in Meissner et al U.S. Pat. No. 4,907,687; Colombo U.S. Pat. No. 4,766,990; Meissner et al U.S. Pat. No. 4,487,307, Smith U.S. Pat. No. 4,010,842 and Paterson et al U.S. Pat. No. 5,413,206 all of which use various conveying techniques to position the stem ends of pears lowermost for transfer to coring machines.

The prior art described above locates either the stem end or blossom end of the produce item and assumes the apple (or pear) is symmetrical about an axis extending from the stem to the blossom, and that the stem and blossom are vertically aligned.

What is needed in this art is a system for orienting produce items that is capable of orienting such items that are asymmetrical about an axis between the stem and blossom.

The present invention achieves that result, and avoids the expense and waste caused by asymmetric produce as described above.

BRIEF SUMMARY OF THE INVENTION

The present invention is a significant departure and improvement over the prior art. One or more depth imaging cameras are utilized together with a programmable, robotic manipulator to create a current three dimensional “characterization” of the surface of an apple (or other produce item) to be oriented. In a first embodiment, the three dimensional “characterization” is a three dimensional model of the surface of the current apple. That current model is then automatically compared with a digitally stored library of three dimensional models of properly oriented symmetric and asymmetric apples (or other produce items) to locate the closest match. The programmable manipulator is then automatically actuated to correct the orientation of the current apple.

In this first embodiment, if the current apple has a “dropped shoulder,” the three dimensional model of that current apple will be compared with properly oriented “dropped shoulder” three dimensional models stored in a digital library accessible by the camera via computer to locate a match. The match model is determined using an Iterative Closest Point (ICP) algorithm. The programmable manipulator is then actuated to correct the orientation of the current apple.

In a second embodiment, preferred for use with apples, a Principal Axis of Curvature (PAC) algorithm is utilized together with a computer, programmable manipulator and one or more depth imaging cameras to create a three dimensional map of the slope for each point on the surface of the apple. The stem and blossom indents of apples have the highest cluster of high slopes, and locating those clusters locates the stem and blossom indents, and the stems and blossoms. Once the stem and blossom indents are located, the manipulator is actuated to correct the orientation of the apple. This embodiment does not require a digital library of stored three dimensional models.

For the first time known to applicants, asymmetric and symmetric apples (or other produce), can be automatically oriented to a desired position as required for subsequent automatic processing such as coring, peeling, etc.

Another aspect of the invention is that bruised apples (or other produce) unacceptable for processing can be identified and separated by the use of color sensitive depth imaging cameras.

A primary object of the invention is to provide a method for automatically orienting asymmetrical, as well as symmetrical, apples and other produce items such a peaches, pears and apricots for further processing such as coring, peeling or pit removal (in the case of peaches and apricots).

A further object of the invention is to reduce the amount of waste, and the expense in dealing with waste occurring in the automatic processing of apples and other produce.

Another object is to increase the yield in the automatic processing of apples and other produce.

A further object is to provide an automatic method of locating and separating bruised apples unfit for processing before any processing has begun.

These and other objects and advantages will become apparent from the description below and the drawings.

DETAILED DESCRIPTION OF THE DRAWINGS

FIG. 1is a schematic cross-sectional representation of a symmetrical apple10having a stem14at upper end11with stem indent13and a blossom end12with blossom indent15and blossom16. The upper indent13contains stem14. The blossom indent15contains blossom16. Axis18extends vertically from stem14to blossom16and is the correct axis along which coring tubes move to remove the undesirable seed cell19. Apple10is symmetrical relative to axis18.

FIG. 2is a schematic cross-sectional representation of asymmetric apple30, since apple30is not symmetric relative to axis38, which extends through stem34and blossom36. The stem indent33is positioned in “dropped shoulder”31. When apple30enters a prior art orientor, the relatively flat blossom end32with indent35is positioned as shown inFIG. 2. The prior art orientors locate indent35and align apple30as shown inFIG. 2, with vertical axis37not extending through stem34and seed cell39. A coring machine removes a cylindrical core along axis37, but misses the seed cell39entirely. This problem is overcome by the present invention.

FIG. 3is an illustration of the asymmetric apple30ofFIG. 2properly oriented by the invention and positioned in a vertical coring device40.

FIGS. 4A-4Bare schematic diagrams of the concept of the invention. Current feed and singulation equipment (not shown) is used to automatically transport individual produce items such as apple105into programmable manipulator100as shown inFIG. 4A. The manipulator100will grab hold of the apple105regardless of orientation and present it to a depth imaging camera, preferably an RGB-D camera200. The RGB-D camera200images each produce item such as apple105repeatedly as the produce item is manipulated and collects color information (Red, Green, and Blue) for every point on the surface of apple105as well as distance of each point from the camera. The manipulator will automatically follow a preset and programmable pattern to manipulate apple105to present the entire surface of the apple105to the camera200. The manipulator's position is tracked using known encoders (not shown).

As shown inFIG. 4A, apple105has a dropped shoulder107and is asymmetric about an axis between its stem106and blossom108. The apple core139is shown between stem106and blossom108. Axis138, extending from stem106to blossom108, is the proper axis to be oriented vertically for transfer to coring and other processing equipment, as noted above. Axis137represents an axis incorrectly located by typical prior art orientors described above. Stem indent133includes stem106. Blossom108is located in blossom indent135. The bottom132of apple105is relatively flat.

Manipulator100has a generally Y-shaped base90which includes a support sleeve91which rotates about axis x-x as support shaft110is rotated. Support shaft is actuated by pneumatic drive means not shown. Sleeve91is formed integrally with and carries shoulders92and93, which in turn pivotally carry arms101and102, at pins103and104. Support shaft110also is connected to, carries and actuates a four bar linkage system including pivotable arms101and102and linkage arms94and95. Linkage arms94and95are pivotally connected to arms101and102by pins96and97and carried by linkage support base96, which in turn is carried by the upper end110aof shaft110(shown in phantom). As shaft110is advanced upwardly inFIG. 4Aby pneumatic or other means, the four base linkage system causes arms101and102to separate, spreading cups121and122apart. As shaft110is retracted downwardly, preferably by pneumatic action, the four bar linkage causes arms101and102to move closer together to cause support cups121,122to contact the apple105without crushing it. As shaft110rotates around axis x-x, the entire manipulator100and apple105are rotated in unison. The rotation of shaft110about axis x-x and the rotation of shafts111and112about axis y-y enable manipulator100to present the entire surface of apple105to camera200. Swivel joints131and132are carried by shafts111and112. Swivel joints carry support cups121and122. Shaft supports141and142are carried at the ends of arms101and102, and support shafts121and122. One small servo motor143is carried by shaft support142. Servo motor143drives both shafts111and112, since shafts111and112are only rotated together when an apple is carried by manipulator100.

Once the stem106and blossom108have been located as described above, the manipulator100is actuated to orient the stem106vertically with respect to blossom108.

FIG. 4Billustrates proper orientation of stem106vertically aligned with blossom108. It is to be understood that the apple105may be properly oriented with the stem106either vertically above blossom108as shown inFIG. 4B, or stem106may be vertically below blossom108.

To orient apple105from the position shown inFIG. 4Ato its proper orientation shown inFIG. 4B, manipulator100is rotated about the x-x axis until stem106and blossom108lie in a plane including the x axis and the z axis (shown best inFIG. 5), wherein the z axis is perpendicular to the plane including the x and y axis. Once the stem106and blossom108lie in the plane including the x and z axes, cups121and122are rotated around the y-y axis to reach the proper orientation shown inFIG. 4B. Once the proper orientation is achieved, cups121and122are separated by support shaft110moving upwardly inFIG. 4Bby a pneumatic drive (not shown) to release the oriented apple to known transfer apparatus for further processing.

It is to be understood that inFIGS. 4A and 4B, camera200and manipulator100are connected to a computer not shown for clarity. The drive mechanism for manipulator100is also not shown for clarity.

In a first embodiment of the invention, images from camera200are used to create a 3D model of the surface of apple105. This 3D model is a “Digital three dimensional characterization” of the surface of the produce item, apple105, being manipulated. An Iterative Closest Point (ICP) algorithm is used to compare the generated model with a previously created pre-existing digital library of a plurality of 3D models of properly oriented symmetric and asymmetrical apples (or other produce items) to locate the closest match between the current generated model and the digitally stored models. The motions needed to correct the orientation of the apple from its current state to a preferred state are calculated. Those motions are then carried out by the manipulator100to produce a properly oriented apple as shown inFIG. 4B. The Iterative Closest Point (ICP) algorithm is known in the art and readily available.

FIG. 5is a perspective and more detailed view of preferred manipulator100shown inFIGS. 4A and 4B, in which shafts111,112shown inFIGS. 4A and 4Bhave been shortened, so that swivel joints131and132are carried by shaft supports141and142. As noted above one small servo motor143(FIG. 4A) is carried by shaft support142and is not visible inFIG. 5. As noted above, manipulator100has arms101and102carried by shaft110which rotates about axis x-x. Support cups121and122are carried by swivel joints131,132carried by shafts111and112. Support cups121,122are utilized to contact and manipulate apple105to achieve correct orientation.

FIG. 6is a block diagram of the control system for said first embodiment is shown generally as50. A known feed system51is actuated to transport an apple to the manipulator100. The manipulator100is actuated to rotate the apple in a preprogrammed sequence stored at52. One or more depth imaging cameras200is actuated to scan the apple repeatedly as it is rotated by manipulator100.

The three dimensional model250of the current apple being scanned is fed into computer300. An Iterative Closest Point (ICP) algorithm260is used to compare the model250with a digital library320of three dimensional digital models of properly oriented symmetrical and asymmetrical apples to determine the closest match and the proper pose for the current apple. The manipulator motions to properly orient the current apple are calculated at330and fed to the manipulator at340. The manipulator is actuated at350to properly orient the current apple, and the apple is then transferred to coring or peeling at360.

FIGS. 7A and 7Bshow an alternative manipulator400that grips the apple using the fin ray effect. Fins410and420are carried by arms401and402, which in turn are carried by a support shaft405which rotates about axis x-x, in similar fashion to manipulator100shown inFIGS. 4A, 4B and 5. The fins410and420grip the produce as shown inFIG. 7B, as known in the manipulator art.

FIG. 8shows a further manipulator500based on an orienting cup520with2wheels530,540off center rather than one rotating wheel in the center. Orienting cup520has an opening (not visible inFIG. 8) formed in its bottom, as known in the art. Apple505is rotated by off-center wheels530and540so that the entire surface of apple505can be repeatedly imaged by one or more depth imaging cameras (not shown inFIG. 8). When the entire surface of apple505has been imaged, either or both wheels530,540are actuated to properly orient apple505.

FIG. 9is a schematic diagram representing a second embodiment of the invention shown generally as700. Four depth imaging cameras711-714are spaced around manipulator740and are repeatedly imaging an asymmetric apple720having a stem734in stem indent733and a blossom736in blossom indent735as apple720is being manipulated. As the apple720is manipulated by manipulator740(such as manipulator100shown and described above the indents)733and736are presented to cameras711-714along with the entire surface of apple720. As the apple continues to be rotated, a three dimensional image of the surface of apple720is fed by cameras711-714to computer760. Computer760is connected to the drive770for manipulator740. Computer760utilizes the known Principal Axis of Curvature (PAC) algorithm to create a three dimensional map of the slope for each point on the surface of apple720. The stem indent733and blossom indent735have the highest clusters of high slopes, which clusters are utilized to locate the indents. When the indents are located, the location of stem734and blossom736are known and the manipulator740is actuated by computer760to properly orient the apple720for processing as described above. Normal orientation for apples aligns stem734vertically with blossom736, with either the stem being above or below the blossom. In using the PAC algorithm, no digital library is required. Although 4 depth imaging cameras are shown inFIG. 10, acceptable results can be achieved with a single camera. A block diagram of the control system for this embodiment is the same as the diagram ofFIG. 6, except that the 3D image at250is of the slopes for each point on the surface of the apple, the algorithm at310is the PAC rather than ICP, and the digital library shown at320is not used with the PAC algorithm.

The Principal Axis of Curvature (PAC) algorithm is known and is not described in detail here. The basic two steps of the PAC algorithm are:1. Represent the segmented apple point cloud in terms of curvature by multiplying surface normal by curvature magnitude. The result is a point cloud that represents the magnitude of surface changes vs. direction. This translates the higher curvature around the stem and blossom indents into a point cloud that is elongated along the stem-blossom axis.2. Estimate the principal axis of the curvature representation of the apple using a robust version of PCA. The principal axis is the axis of maximum variability. The principal axis, or first principal component, represents the estimated axis of the apple. Perform a few iterations of principal axis estimation with outlier removal.

As shown in the diagram ofFIG. 10, the most preferred embodiment of the invention utilizes four RGB-D cameras810,820,830and840placed around manipulator100, shown inFIGS. 4A, 4B and 5. Computer860is connected to manipulator100and all four cameras810,820,830and840.

One or more depth imaging cameras may be utilized. The RGB-D cameras are preferred, since they also provide color information. The color information is utilized to detect dark or discolored regions on the surface of bruised apples which are not appropriate for automatic coring or peeling. Such bruised apples are separated and either discarded or processed by alternate means.

As used herein and in the claims, the phrase “digital three dimensional characterization of the surface” refers to any useful digital depiction, model or representation of the shape of the surface or of any characteristic of the surface such as slope.

As used herein and in the claims, the phrase “depth imaging camera” refers to any camera capable of generating three dimensional images or characterizations of the surface of an object within the view of said camera.

It is to be understood that locating the stem and blossom is done in most instances in the case of apples by locating the stem indent and/or blossom indent using the PCA algorithm and assuming that the stem and blossom are located at the center of each respective indent. Accordingly, as used herein and in the claims, the phrase “locating the stem and blossom” is used broadly to include locating the stem indent and blossom indent. In addition, for many varieties of peaches, pears and apricots where the stem and/or blossom indents may be too small to use the PCA algorithm, the first embodiment using a digital library and the ICP algorithm would be the appropriate method.

In the case of peaches and apricots, the proper orientation is required for removing the stone or pit. The present invention may be utilized to locate not only the stems and blossoms of peaches and apricots, but also the “suture line” of these items. Proper orientation of the suture line is significant in removing the stone or pit, as is known in the art.