Source: https://patents.google.com/patent/US9743988B2/en
Timestamp: 2018-03-20 22:39:03
Document Index: 233820129

Matched Legal Cases: ['Application No. 11851566', 'Application No. 11851566', 'Application No. 201180080959', 'Application No. 2013', 'Application No. 2013', 'Application No. 10', 'Application No. 2013', 'Application No. 2015', 'Application No. 2011349503', 'Application No. 2', 'Application No. 11851566', 'Application No. 11851566']

US9743988B2 - Methods and systems for directing movement of a tool in hair transplantation procedures - Google Patents
US9743988B2
US9743988B2 US14538362 US201414538362A US9743988B2 US 9743988 B2 US9743988 B2 US 9743988B2 US 14538362 US14538362 US 14538362 US 201414538362 A US201414538362 A US 201414538362A US 9743988 B2 US9743988 B2 US 9743988B2
US20150066054A1 (en )
FIGS. 4(a)-(f) show various examples of implementations of the methodology according to an embodiment of the invention.
FIGS. 4(g)-(h) demonstrate an example of an embodiment according to “an exclusion zone” methodology.
FIGS. 5(a)-(g) show an example of identifying and recording fiducials as could be implemented in an example of the embodiment of the invention.
FIGS. 6(a)-(f) show another example of implementation of the methodology according to an embodiment of the invention.
FIGS. 7(a)-(f) illustrate examples of various selection criteria according to various embodiments of the invention.
FIGS. 8(a)-(f) are schematic representations illustrating an example of the use of satellite sites in the provision of exclusion zones.
FIGS. 9(a) and (b) illustrate the difference between using and not using the satellite exclusion zone methodology.
According to the various embodiments described herein, a variety of methodologies and systems are provided which enable a tool to automatically proceed from where it left off prior to an interruption that the procedure may be subject to, continuing its operation and essentially providing a seamless operational procedure. The systems and methods described herein enable the tool to maintain its direction of travel over the patient's body surface that it had despite patient's movement or other interruptions, to recognize where it has previously harvested follicular units or implanted them, and continue to travel in that general direction to harvest or implant further follicular units. The inventions described herein enable the system to operate in a fully-automated fashion, if desired, without requiring relocation of the base of the robotic system, relocation of the body surface, physician assistance or human intervention. In addition, the present invention provides methodologies that enable a tool operated by an automated system or under computer control to be operated to change its direction of travel when required, without necessarily requiring human intervention, although a user could override any automated movement if desired.
At step 115, a processor or an image processor, an example of which is described later in reference to FIG. 2, processes and records an identity and a location of each of the fiducials in a frame of reference of an image acquisition device (e.g., in a camera field of view). Such initial recording of fiducials could be referred to as “fiducial registration.” The fiducials could be recorded in various coordinate systems, for example, in a fixed “world” coordinate system. In the example of FIGS. 4(a)-4(f), the fiducials are described as recorded in a coordinate system fixed to the camera. In situations in which an image acquired by the image acquisition device includes only a subset of the fiducials such that images of additional fiducials are needed, step 120 provides for acquiring additional images as needed, for example, including other subsets of the fiducials, until all fiducials have been identified. (This aspect will be described in greater detail with respect to FIG. 5). In an optional step 125 (shown in dotted line), based on the location of the each of the plurality of fiducials, a boundary of an area, such an area within which hair grafts or follicular units are intended to be harvested from or implanted into, may be determined. The boundaries may be determined automatically, for example, by drawing lines between various fiducials. The boundaries may be also adjusted to eliminate certain portions of the bound area where harvesting or implantation is difficult, as explained and described in further detail in reference to FIG. 3. In order to accommodate for patient motion, temporary interruptions, and any other incident that may cause a shift in location of the fiducials in the camera reference frame, as often as required (as may be determined by the user), updated images of the body surface are acquired, the images containing an image of the plurality of fiducials or a subset thereof. Due to patient motion, or another such temporary interruption, the locations of the fiducials in these updated images may be in a revised location with respect to the frame of reference of the image acquisition device. The processor in step 130 processes the revised location of each of the plurality of fiducials in the frame of reference of the image acquisition device, the revised locations of each of the plurality of fiducials which may be different from the locations previously processed. Having acquired the revised locations of the fiducials, and with the knowledge of the original locations of the fiducials, an offset for at least some or all of the fiducial locations may be determined in step 130. Based on this offset information, the processor also in step 130 may process revised locations for each of the locations of interest, such as locations from which follicular units have already been harvested (if harvesting has already started in a region of interest within the boundary) or into which follicular units have already been implanted (if such implanting has been started). Optionally, step 130 may also comprise determining the revised boundary, for example, of the harvesting/implanting area based on the revised locations of the fiducials. However, it is not necessary in some embodiments to determine the whole revised boundary as this information may be automatically ascertained simply based on the offset of the minimum number of the fiducials. In reference to the example of hair transplantation, having determined the offsets, and with the knowledge of the locations of the follicular units that have been harvested or implanted (if any) with respect of the fiducials, it is possible in step 135 to determine or select a location from where the next hair follicle is to be harvested such that hair follicles are not taken from an already harvested location, or determine a location into which the next hair follicle is to be implanted such that hair follicles are not implanted into locations into which hair follicles have already been implanted. Such selection may be made using a processor programmed to perform the above-described step, such as a processor described in reference to FIG. 2. In step 140, hair graft or follicular unit is harvested from or implanted into the selected location. When the next hair follicle is harvested or implanted, the location from where it has been harvested from, or implanted into, may be registered or recorded by the processor in step 145. This registration may include information on the location of the harvest or implant with respect to at least one of the plurality of fiducials, or the determined boundary. Optionally, in step 150, the method may comprise creating and displaying a virtual representation on the image of the location from which the follicular unit has been harvested (or at least dissected from the surrounding tissue for further removal using forceps or vacuum), or the location into which a follicular unit has been implanted. Such visual representation, for example, on a monitor (e.g. a computer screen) is especially beneficial for the user to easily and quickly identify locations where hair grafts have been dissected or harvested, and also to differentiate between the previously existing follicular units and the newly implanted ones. The visual representations of step 150 may be implemented by using different colors, shapes or other appropriate differentiating features. In step 155 the processor determines, based on the information is has recorded with respect to the area and the locations of the follicular units that have been harvested or implanted, if follicular units have been harvested from all desired sites, or if follicular units have been implanted into all desired sites. In the event that all follicular units have been harvested or implanted, the processor may communicate this information, for example, to the image acquisition device. In addition, the processor may communicate this information to the user, typically providing an indication to the user (via the monitor, voice command, or any other appropriate technique), for example, that step 110 may begin again at a new donor or recipient region. In the event there are still follicular units to harvest or implant, the processor continues to repeat steps 130-155 until all desired follicular unit are harvested or implanted. For example, updated images with the updated fiducial information are processes, offsets determined, the next harvest site or implant site is selected, etc. In this manner, a methodology is provided to enable hair follicles to continue to be harvested from or implanted into a body surface in a continuous and automatic fashion despite potential patient movements and interruptions. The tool is able to be moved to each new harvesting or implantation location with respect to fiducials, the fiducials providing a mechanism of recognizing the location of the harvesting/implanting area on the body surface, despite movement of the patient, or the image acquisition device.
FIGS. 4(a)-4(f) illustrate how the steps of FIG. 1 can be implemented utilizing the system of FIG. 2 and fiducials, for example, similar to those as illustrated in FIG. 3. In FIG. 4(a), a tensioning device 400 is illustrated. In order to more easily explain the various methods of the current invention, rather than utilize fiducials such as those illustrated in FIG. 3, the discussion will utilize fiducials 405 which are illustrated as a series of alphabetic characters, A to F, along the vertical side sections of the skin tensioner, and a set of numeric characters, 1 to 8, along the horizontal side sections of the skin tensioner. The fiducials generally may be placed in arbitrary positions relative to a working area. As mentioned above, the fiducials are placed such that a known feature of the fiducial, for example the center of each fiducial, or a known boundary of the fiducial is at a known distance from inner bounding edge of the skin tensioner. For example, the fiducials may comprise circular shaped adhesive labels that affix to the tensioning device, the edges of the circular shaped adhesive labels being such that the size of the label is less than the size of the vertical and horizontal structures of the tensioning device to which it is attached, so that when placed on the tensioning device, the fiducials themselves may be located, for example, in the region of 2 mm or so from the inner boundary of the tensioning device. It is desirable to position fiducials (whether it is on the tensioner, or on a skin itself) or locate natural fiducials, such as follicular units, such that the relative position of the fiducials do not change, or if they do, they do not change significantly during the procedure. The system, in particular the processor, can process images acquired by the image acquisition device to detect substantial relative motion, for example motion in the region of 1 mm in a field of view in the region of 50 mm, that may be caused, for example, by misidentification or detachment of a fiducial (if it is an externally placed fiducial), and report such an error so that measures may be taken if necessary to compensate or correct for the error. Of course, the surface on which the fiducials are located is free to move (e.g. patient moving his/her head or getting up). The translation and rotation of the surface with the fiducials due to motions can be computed any time when at least three non-collinear fiducials are visible. When the shifted and/or rotated fiducial locations are detected (for example, by an imaging device and/or image processing software or hardware which may form a part of an imaging system), the procedure is able to continue at the next harvest site. If fewer than three non-collinear fiducials are visible, another attempt to image and register fiducials is made so that three or more fiducials become visible, as described in more detail in reference to FIGS. 5(a)-5(g).
According to the methodology of an embodiment of the invention, and with reference to FIG. 4(a) the location and optionally the orientation of each of the fiducials 405 may be identified, registered, and electronically saved via the image processor. In addition, the user may also specify via an input device such as the keyboard or the mouse, information pertaining to the physical parameters of the tensioning device 400 (if such tensioning device is used) and the tool, information such as the height of the tensioning device 400 relative to the body surface and the diameter of the tool for example. Based on this information, the processor may, optionally, determine the location and orientation of a revised boundary 410. To aid in the understanding, it will be assumed that the fiducials 405 are recorded in the coordinate system of the image acquisition device, which as indicated above is on the robotic arm, though they may be recorded in any appropriate reference frame.
FIGS. 4(a)-4(f) will be discussed, as an example, in reference to the hair harvesting, and assuming that revised boundaries 410 are utilized. However, it should be understood that this description applies and could be adjusted accordingly in reference to hair implantation, and to eliminate the determination of revised boundaries 410. As illustrated in FIG. 4(a) the harvesting tool is operated to initiate the harvesting procedure from the bottom left hand corner of the quadrilaterally shaped bound area, bounded by the revised boundary 410. During the hair transplantation process, often various fluids, including for example, blood and saline will be present on the body surface. It was discovered that it is advisable, especially in the computer-implemented or robotic hair transplantation procedures, that the harvesting or implantation process begin from the bottom of the frame, whether it be in the right or left corner. This way any appearing blood or other fluid will tend to flow downwards, and therefore, will less likely compromise the image of the potential subsequent hair harvesting or implantation sites, thus optimizing any image processing that may be implemented. The harvesting tool may be moved to the initial or first harvesting location, such as location 415 in FIG. 4(a), directly or indirectly by the physician (for example, the physician may click on the image to identify the desired harvesting location), or the processor may be configured or programmed to find this location itself, for example, based on the information it has acquired on the fiducials, the processor may then provide instructions to the control unit to move the tool accordingly.
As indicated in FIG. 4(b) the processor can be configured to operate the tool to harvest follicular units at predetermined locations, such as in this example, by passing over, for example, any locations in row C-F between fiducial locations 5 and 7 without harvesting a follicular unit, and making its subsequent or second harvest at the location 425 that could be defined as C7. It should be understood that the selected harvesting location, for example, on the row C-F does not have to be exactly at the level of the location of the fiducials 405 (such as fiducial 7), but rather may be anywhere and at any distance from a particular fiducial (e.g., between the level of fiducials 6 and 7). When the processor determines that the tool is within a predetermined distance from fiducial F and the end of this first harvesting row, or that the revised boundary 410 has been reached, the processor provides instructions to the control unit to cause the tool to move in a direction away from fiducial F to automatically increment to the next harvesting row. In this particular case, as illustrated in FIG. 4(c), the tool is controlled such that it moves initially in an upwardly direction 430, from F to E, and then in a direction denoted by arrow 435, away from E, away from the revised boundary 410, and along the virtual line EB. In this case, the tool is controlled to move to a harvesting site 440 located at approximately B-8, and operated to harvest a follicular unit at that location before moving on. This procedure can continue without requiring intervention from the operator or physician. It should be understood, however, that the operator may intervene at any time to override an automated movement and select a different follicular unit to be harvested, if desired or necessary. The system is configured to direct a tool to move and operate, for example, at least in part based on the location of the fiducials. In this manner, the tool can be operated to turn automatically when the revised boundary is approached and start the next row of the harvesting process, and to stop automatically when the area bound by the revised boundary 410 has had all desired follicular units harvested.
According to another example of implementation according to the inventions described herein, FIG. 4(d) indicates a location 445 denoted by an “X”, at which it is intended that the next follicular unit be harvested. However, let us assume that for whatever reason, there is an interruption, perhaps the patient moves, either temporarily leaving the operation chair, or just shifting to get in a more comfortable position. Even though the patient has moved, the view seen by the camera which is in this instance located on the robotic arm, will be substantially unchanged in the global context, that is, the view relative to the chair will be the same (assuming that the chair is not moved with respect to the robot). However, the view with respect to the patient's body surface may be different. As indicated in FIG. 4(e), the patient's body surface may have moved such that the existing follicular units can be seen to have moved both to the right, and upwardly, in the frame of view 450. If the tool was moved to harvest the next follicular unit at a location with coordinates referenced with respect to the frame of view 450, it can be seen that the follicular unit would be harvested from a location 455 marked with the “X” which is not the original desired location marked as 445 in FIG. 4(d). As indicated in FIG. 4(e), this location 455 is close to another follicular unit 460 that could be damaged by harvesting the location 455. This location 455 is also close to a location from which another follicular unit has already been harvested, location 465. Moreover, this would not maintain the intended row-to-row spacing of harvested follicular units, and does not maintain the intended spacing of the harvested follicular units from one another. By registering the intended harvesting site with respect to the fiducials 405, the robotic system is able to avoid some or all of these errors, and additionally is able to continue the harvesting process without necessarily requiring significant intervention of the physician to do so. The robotic system is configured to determine the location and orientation of each of the fiducials 405, and compare these new or revised locations and orientations with the already saved information on each of the distinctive fiducials 405. For example, in this particular case, it will determine that the location of each of the fiducials 405 has moved a certain distance towards the right hand side of the frame of view, and a certain distance in an upwardly direction also. Using fiducial tracking techniques that are known in the art, the system is able to determine how each of the initially identified fiducials has been transformed in location and orientation, and determine the transformation that needs to be applied to the location 455, to relocate that same location 455 with respect to the fiducials 405. Having acquired this information, the processor is then able, using known transformation techniques, to modify the location and orientation information of the intended implantation location 455 accordingly, applying the necessary transformation of the coordinates of the location, so that the tool can be operated to move to the correct harvesting site 470 (which correspond to the originally marked site 445), as illustrated in FIG. 4(f). In this manner, the system is configured to operate the follicular unit harvesting tool to maintain its harvesting direction that is along the virtual row A-D, despite patient movement. In addition, the system is configured to ensure that harvesting does not occur at sites where harvesting has already taken place, enabling site to site spacing and row-to-row spacing to be maintained. To this end, the provision of visual image, for example, in the form of circles centered about where harvesting has been performed, provides the user with a visual representation that confirms that harvesting has occurred at the site to site spacing and row-to-row spacing desired. Obviously, should the visual circular representations not correspond with the desired outcome, the user has an opportunity, based on the recognition of the deviation from expectation, to correct for any errors that may be visualized. This correction can be implemented at any moment that deviation from expectation has been detected, and multiple corrections may be performed throughout any procedure. In this manner, the system is able to harvest follicular units despite patient movement. It will be appreciated that although the above has been described with respect to the harvesting process, the methodologies described above can be easily adapted to apply to the implantation process, or other procedures.
FIGS. 4(g) and 4(h) illustrate an example of an embodiment using an exclusion region methodology. Turning first to FIG. 4(g), already harvested locations 472 and 474 are identified with circular representations 476 and 478 indicating larger regions centered about the harvest locations 472 and 474. These circular representations 476 and 478 provide a visual image of areas from which it is not desirable to harvest additional follicular units, as these would be too close to already harvested follicular units. The processor may select a potential harvesting location 480, which is outside the circular representations 476 and 478. In addition, the processor creates an arbitrary shaped feature 482, which in this instance is shaped as a tear-drop, around the potential harvesting location 480. As can be seen in the figure, the shape of the feature 482 can be described as a circle around the potential harvesting location 480, which has been extended, or stretched in the direction 484, which is the direction in which the tool is both travelling and angled to harvest, thus forming a tear-drop shape (or exclusion zone). Having created this exclusion zone 482, the processor determines whether there are any already harvested sites that fall within the exclusion zone 482 in addition to the potential harvesting site 480. In this instance, the already existing harvesting site 472 can be seen to fall within the exclusion zone of the tear-drop 482, and so the processor will determine that this potential harvesting site 480 is not a site from which a follicular unit should be harvested. Harvesting a follicular unit from this location 480, with the tool orientated at the selected angle and in the direction 484, could possibly create a harvesting path that coincides or intersects with the already existing path that was created at location 472. Therefore, the processor selects an alternative potential harvesting site, for example that indicated in FIG. 4(h).
In FIG. 4(h), once again already harvested locations 472 and 474 are identified with circular representations 476 and 478 indicating larger regions centered about the harvest locations 472 and 474, from which it is not desirable to harvest additional follicular units. The processor selects a potential harvesting location 486, which is outside the circular representations 476 and 478. In addition, the processor creates an arbitrary shaped feature 490 (or exclusion zone) which once again is shaped as a tear-drop, around the potential harvesting location 486. As can be seen in the figure, the shape of the feature 490 can be described as a circle around the potential harvesting location 486, which has been extended, or stretched in the direction 488, which is the direction in which the tool is both travelling and angled to harvest, thus forming a shape that is tear-drop shaped. Having created this exclusion zone 490, the processor determines whether there are any already harvested sites that fall within the exclusion zone 490 in addition to the potential harvesting site 486. In this instance, while there are follicular units within the exclusion zone 490, for example, the harvesting site 492, none of them has been harvested yet. Therefore, a candidate or potential harvesting site 486 is an acceptable candidate, and the processor may determine that this potential harvesting site 486 could be harvested without intersecting with any already existing harvesting path.
The generation of visual representations that define exclusion zones that are centered, for example, about a harvesting site, may create an image that has numerous overlapping representations, and consequently an image that has numerous gaps formed between each of the distinct exclusion zones. This is illustrated in FIG. 8(a) in which already existing procedure sites, such as harvesting sites 805, 810 and 815, each have associated exclusion zones 820, 825 and 830 respectively, created as described hereinbefore. The exclusion zones 820, 825 and 830 create a gap 835 as illustrated. Gaps such as these tend to create a rather “unfriendly” visual representation for the user and the processor. An image with a vast number of these gaps can make it difficult for the eye to easily identify or focus on the “larger” gaps, and may also consume additional processing time by the processor. By avoiding the creation of these gaps, and in particular the relatively small gaps, a visual representation that is more pleasing to the eye can be created, a more “friendly” visual representation for the processor and/or the user, one in which gaps are fewer and easier to identify. This is particularly beneficial, for example, in situations where manual follicular unit selection is desired, situations in which for example, the user can manually select follicular units to be harvested that were missed by an automatic selection algorithm or close to the skin tensioner but still harvestable in the eyes of the user.
One way in which the gap 835 illustrated in FIG. 8(a) can be avoided, is by using an alternative or additional methodology which serves to fill the gaps between the exclusion zones 820, 825 and 830. In one embodiment, more than one existing follicular unit harvest site is used to create a visual representation of the exclusion zone for that particular harvested follicular unit. For example, in one such method for creating a visual representation of the exclusion zone, the visual representation is generated by using not only the newly harvested follicular unit, but by using information from its neighboring or satellite sites, the sites of previously harvested follicular units that are in close proximity to the newly harvested follicular unit.
FIG. 8(b) shows the two existing harvested follicular unit sites 805 and 810, and a site 815 which represents a site of a newly harvested follicular unit. For ease of explanation, the exclusion zones 820, 825 and 830 have been omitted. When processing the exclusion zone for the newly harvested follicular unit site 815, the processor is configured to determine whether or not the already existing follicular unit harvesting site 805 is within, for example, a predetermined distance from it. This predetermined distance may be based, for example, on a multiple of the minimum harvest distance described above, the multiple being greater than one, and ranging, for example, from 1.5 to 2.5. For example the predetermined distance may be less than or equal to at least twice the minimum harvest distance from the newly harvested follicular unit site 815, that is a distance of, for example, 3.8 mm. If it is found that the already existing follicular unit harvesting site 805, for example, is less than or equal to at least twice the minimum harvest distance away from the newly harvested follicular unit site 815, the already existing follicular unit harvesting site 805 may be considered to be a satellite site with respect to the newly harvested follicular unit site 815. If, however, it is found that the already existing follicular unit harvesting site 805 is more than at least twice the minimum harvest distance away from the newly harvested follicular unit site 815, the already existing follicular unit harvesting site 805 may be considered not to be a satellite site with respect to the newly harvested follicular unit site 815. In a similar manner, when processing the exclusion zone for the newly harvested follicular unit site 815, the processor is configured to determine whether or not the already existing follicular unit harvesting site 810 is less than or equal to at least twice the minimum harvest distance from the newly harvested follicular unit site 815, that is a distance of, for example, 3.8 mm.
For ease of understanding, let us assume that both existing harvested follicular unit sites 805 and 810 are less than or equal to at least two times the minimum harvesting distance from the newly harvested follicular unit site 815. In this instance the processor creates a closed loop profile, or a supplemental exclusion zone, based on the locations of the newly harvested follicular unit site 815, and the existing harvested follicular unit sites 805 and 810, forming a triangular shape 840 as illustrated in FIG. 8(b). The processor combines or superimposes this closed loop triangular profile 840 (the supplemental exclusion zone) onto the three circular exclusion zones 820, 825 and 830, as illustrated in FIG. 8(c) to form a visual presentation of the modified exclusion zone 845, as illustrated in FIG. 8(d) that no longer incorporates the relatively small gap 835.
FIG. 8(e) illustrates an example in which a newly harvested follicular unit site 850 is introduced, sites 805, 810 and 815 all being already existing harvested follicular unit sites. If one assumes that the existing harvested follicular unit sites 805, 810 and 815 are less than or equal to a certain minimum harvesting distance, for example, at least two times the minimum harvesting distance, from newly harvested follicular unit site 850, they will all be considered satellite sites to newly harvested follicular unit site 850. The processor in this instance is configured to create a closed loop profile (supplemental exclusion zone), based on the locations of the newly harvested follicular unit site 850, and the existing harvested follicular unit sites 805, 810 and 815, forming a polygon with indices 805, 810, 815 and 850. However if, for example, it is determined that only existing harvested follicular unit sites 810 and 815 are less than or equal to at least two times the minimum harvesting distance from newly harvested follicular unit site 850, and existing follicular unit harvesting site 805 is more than at least two times the minimum harvesting distance from the newly harvested follicular unit site 850, only existing harvested follicular unit sites 810 and 815 will be considered satellite sites for site 850, and the close loop profile will be a triangle (not shown) with the indices of 810, 815 and 850.
Finally, FIG. 8(f) illustrates an example in which the centrally located newly harvested follicular unit site 885 has five satellite sites 860, 865, 870, 875 and 880 around it. Rather than providing a visual representation of an exclusion zone that only comprises a simple circles surrounding the newly harvested follicular unit site 885, the processor, having determined that each of the already existing harvested follicular unit sites 860, 865, 870, 875 and 880 surrounding the centrally located newly harvested follicular unit site 885 are less than or equal to, in the provided example, twice the minimum harvesting distance from it, forms a polygon 890 (supplemental exclusion zone) linking all the satellites to surround the centrally located newly harvested follicular unit site 885. In this instance, by combining this polygon shape 890 with the six circular exclusion zones corresponding to the already existing harvested follicular unit sites 860, 865, 870, 875 and 880, no gaps are visualized within the visual representation of the modified exclusion zone. This potentially enables a reduction in computer processing time, and it also provides benefit to the user by ultimately enabling the user to more easily identify on a display and focus on the larger gaps that may exist.
FIG. 9(a) illustrates a visual representation which only utilizes a methodology in which the exclusion zones are centered about the harvested follicular units. In FIG. 9(a) attention is drawn to one particular gap 910. On the other hand, FIG. 9(b) illustrates the visual representation which utilizes a combined methodology in which the exclusion zones are centered about the harvested follicular units and then combined with the overlay of the supplemental exclusion zone provides by the appropriate satellite sites. As seen in FIG. 9(b), the location where there used to be a gap 910 is no longer there. A visually more “friendly” image has been rendered. In summary, according to some embodiments a method is provided for generating a visual representation of a region where a procedure was performed. The method comprising generating a visual representation of one or more procedure sites (e.g. harvest sites) where a procedure was performed. If more than one procedure site exists, the visual representation of the exclusion zones for each of the procedure sites are overlapped. The method further comprises generating one or more supplemental exclusion zones for any or all procedure sites; and overlapping the exclusion zone and the supplemental exclusion zones to generate the representation of the performed procedure region. The step of generating one or more supplemental exclusion zones may be accomplished by comparing a distance between a particular procedure site and one or more previous procedure sites that surround that particular procedure site, and for those surrounding sites where the distance is within a predetermined or selected limit, such surrounding sites are identified as the satellite sites for the particular site. In some embodiments, the above-mentioned comparison may be run against every existing procedure site to identify a corresponding collection of the satellite sites for each existing procedure site. In some embodiments, with reference to hair transplantation, each new harvested hair graft may be compared to any or all other previous harvested hair grafts and be added, as appropriate, based on the results of the comparison, to a collection of the satellite sites for each relevant previous harvest site. Alternatively or in addition, any or all previous harvest sites may be compared to a newly harvested site and, based on the results of such comparison, be identified as the satellite sites for the newly harvested site. In some embodiments, the satellite sites may be sorted based on certain criteria, such as the tangle angle in the coordinate system, for example, with the smaller angle going first, forming a counter-clockwise sequence, or with a greater angle going first, forming a clockwise sequence. This sorting may be used to generate a continuous convex profile. Without sorting, the random sequencing of the satellite sites may miss some parts of the geometry profile. In some embodiments, the method may further comprise updating and displaying the previous procedure region.
It may be desirable in various procedures to identify “reserved regions” where procedure should not be performed. These reserved regions will be described in reference to hair harvesting and implantation and therefore will be referred to the “reserved harvest regions”, however, it should be understood that this description applies to various “reserved regions” for various medical procedures within the scope of the inventions described herein. Reserved harvest regions define areas from which hairs are not to be selected for harvesting. These reserved harvest regions may define areas where skin conditions exist that make the area unsuitable or undesirable for harvesting from or implanting into, areas which contain previously implanted follicular units, areas containing a particular classification of follicular unit (as F1 for example) that are not desired for the current harvest, areas where moles or scars exist, or define areas exhibiting any number of other conditions. These reserved harvest regions can be illustrated, as shown, for example, in FIG. 10, as a box 1010, or as a circular representation (note that if a circular representation is used, the circles representing the reserved harvest regions may be formed in a different color than the circles used to identify the exclusion regions 1020), or as any arbitrary shape, and may be created in numerous ways. For example, the user may manually define a reserved harvesting site by manually clicking a mouse at a point within the revised boundary 410, to create a reserved harvest region box of pre-defined size. Alternatively, a reserved harvest region box may be created whose sides may be adjusted by the user, or several points may be identified by the user, and the processor may create a closed-loop arbitrary shape that encompasses all the identified points. In an alternative, the reserved harvest regions may be automatically created by the processor once it has processed the information contained in the acquired images, and the user may be allowed to accept or reject these automatically created reserved harvest regions. It will be apparent that there are many other ways in which such reserved harvest regions may be created for or by the user.
According to another aspect of the present application, examples of locating and registering a plurality of fiducials are described in reference to FIGS. 5(a)-5(g). With reference to the example of the robotic system of FIG. 2, since the field of view of the cameras (for example, 2 cameras used for stereovision) may be smaller than the area spanned by the fiducials, it may be necessary to move the robot around the boundary (e.g. perimeter of the tensioner or a region defined by a pattern of existing hairs acting as fiducials) to capture the locations of the fiducials. This motion may be performed, for example, manually by an operator's hand dragging the imaging mechanism attached to the robotic arm using “force-control”, or by manipulating the robotic pendant. However, in a preferred embodiment, a robotic arm with the attached image acquisition device may be moved automatically around the periphery of the skin tensioner (or around the boundary with a plurality of fiducials that defines the harvesting or implantation region). In the automated approach, the robot may be first moved manually to the initial position which brings enough fiducials into view to establish the fiducial frame of reference. Typically, it requires at least three (3) fiducials. FIG. 5(a) illustrates an initial frame of view 502 taken from the frame of view of the camera(s) which is mounted on a robotic arm of a follicular unit harvesting or implanting system, the frame of view having a center point of reference 504. In this embodiment, four fiducials A, B, 1 and 2 are visible in the initial frame of view 502. In order to utilize the teachings of this invention, the system has to acquire the location and/or orientation of each fiducial with reference to at least one other fiducial. As mentioned above, in order to obtain information pertaining to both location and orientation, at least three fiducials are required to be visible in the initial frame of view 502. For example, some examples of obtaining, tracking and recording information about fiducials that could be used in the present invention is described in the commonly owned co-pending patent application published as US 2010-0080415A1 on Apr. 1, 2010, which is incorporated herein by reference.
To enable the system to acquire the location and optionally the orientation of the other fiducials, the system initially moves the field of view of the camera over the body surface such that one of the fiducials that was in the initial frame of view 502, is located at the center of the frame of view, that is, that the centroid of fiducial 1 is substantially aligned with the point of reference 504, as shown in FIG. 5(b). This first fiducial 1 is allocated location and optionally orientation coordinates, for example it may be given the reference coordinates of (2,4). The image processor subsequently identifies the next closest fiducial that has not already been centered. In the event that there are two or more closest fiducials, the system is configured to select the closest fiducial according to a predetermined selection mechanism. The selection mechanism may be relatively simple, such as always selecting the one towards a particular direction, and only doing so if the reference coordinates of the fiducial in that direction have not already been acquired. In this instance, the selection mechanism hierarchy may comprise, for example, the order of to the right, downwards in direction, toward the left and finally upwards in direction.
FIG. 5(c) illustrates the camera having been moved over the body surface such that the centroid of the closest fiducial 2, to the right hand side of the fiducial 1 is located at the reference location 504 of the field of view 502. In order to get to this particular location, the movement undertaken by the camera itself is registered. For example, if the camera movement from a position where the centroid of fiducial 1 was at the point of reference 504 to a position where the centroid of fiducial 2 was at the point of reference 504, is defined by (2,0), then the coordinate for the location of the centroid of fiducial 2 would be (4, 4). Similarly, in FIG. 5(d) the camera movement a position where the centroid of fiducial 2 was at the point of reference 504 to a position where the centroid of fiducial 3 was at the point of reference 504, as (2,0), then the coordinate for the location of the centroid of fiducial 3 would be (6, 4). It can be seen that in FIG. 5(e), once the centroid of fiducial 4 is placed at the point of reference 504, and allocated the coordinates (8,4) there is no fiducial to the right, and in FIG. 5(f), the camera is moved (2,−1) such that the centroid of fiducial D is at the point of reference 504. In this manner, the location of the centroids of all fourteen distinguishable fiducials (shown in FIG. 5(g)) are known with respect to each other. In possession of this information, in some embodiments virtual lines 506, 508, 510 and 512 may be drawn to define an initial boundary, and after taking into account the location of the centroids of the fiducials, for example, from the inner edges of the tensioning device, the relative height of the tensioning device above the body surface (assuming an embodiment where there is one) and/or the tool diameter, a revised boundary 514 can be determined, inside of which the coordinates of follicular unit can be identified for harvesting, or the coordinates of follicular unit implantation sites can be identified for implantation.
FIGS. 6(a)-6(f) demonstrate an embodiment according to another implementation of the invention which uses a defined virtual selection region. The tool may be moved automatically within such selection region as explained below based on desired criteria. Let us assume, for the purposes of this particular discussion, that the revised boundary 610 has dimensions, for example, in the region of 4 cm horizontally and 3 cm vertically. Having established coordinates of the fiducials (such as fiducials 605 illustrated in FIG. 6a ), as described in earlier examples, the tool is operated (for example, automatically or semi-automatically) to initiate the harvesting procedure from the bottom left hand corner of the revised bound area. For example, the tool may be operated to move to the location that is approximately at the intersection of row C-F and column 1-5, and aligned with the follicular unit 615. The processor at this time may also dictate that the tool be moved in the general direction of arrow 620, away from the location of fiducial C and towards fiducial F, in a horizontal direction substantially parallel to a horizontal side of the revised boundary 610. Based on the exact coordinates of the tool's location with respect to the image frame of reference, the processor may compute virtual boundaries of a smaller virtual selection region 625 located just in front of the tool in the direction of travel 620. In this particular illustrated example, the virtual selection region 625 may comprise a quadrilateral, such as a rectangle having, for example, dimensions of 6-8 mm by 3.5-4.5 mm. Other dimensions of the selection region 625 are also contemplated within the scope of this application. Use of a smaller virtual selection region 625 reduces the computation required to find a subsequent follicular unit to harvest by restricting the area of consideration to an area just in front of the previous harvested follicular unit 615 and along the direction of travel 620. The tool is operated to harvest the follicular unit 615 and the location of the harvested follicular unit 615 is visually identified, for example, by a circle 630, as seen in FIG. 6(a). After harvesting the follicular unit 615 the harvesting tool is moved in the general direction of the arrow 620, and operated to harvest one or more follicular units located within the virtual selection region 625. As seen, there are several follicular units located within the region 625. However, the next selected follicular unit for harvesting may be not the follicular unit located within the shortest distance from the harvested follicular unit 615 inside the region 625 (such as follicular unit 640), but may instead be based on predetermined selection criteria, such as in this example where the tool is moved to the location of the follicular unit 635 that is the closest to the horizontal boundary 610.
Examples of a few criteria that could be used in directing movement of the tool within the selection region (such as region 625 of FIG. 6(a)) are described with reference to FIGS. 7(a)-7(e) below, but it will be appreciated that many other criteria may be chosen or predetermined, and such criteria may vary during the hair transplantation process. In FIGS. 7(a) and 7(b), three follicular units have been identified within the selection region 625, follicular units 705, 710 and 715. One particular selection criteria may be, for example, such that if follicular unit 705 is harvested, the system will be operated to harvest follicular unit 715, and leave 710 un-harvested, effectively harvesting every other follicular unit. In an alternative, or an additional predetermined selection criteria, as illustrated in FIGS. 7(c) and 7(d), if follicular unit 720 is harvested, the predetermined selection criteria may be to harvest every other follicular unit within the selection region except when the distance to the next available follicular unit exceeds certain predetermined distance. In the example of FIGS. 7(c) and 7(d), even though follicular unit 725 is the next available follicular unit, it is harvested because it is located at a distance, for example greater than 1.9 mm away, from the already harvested follicular unit 720. As seen in FIG. 7(b), once the follicular unit 725 has been harvested, a new virtual selection region 625 is created next to the harvested follicular unit in the same direction of travel. Turning now to FIGS. 7(e) and 7(f), in this illustration, once follicular unit 730 has been harvested, follicular unit 735 is left un-harvested, and although follicular unit 745 is the next available follicular unit in the horizontal direction, it too is left un-harvested. In this example, the predetermined selection criteria is set such that the next follicular unit available can be selected whether it be the next available closest in the horizontal or the vertical direction, provided that it is contained within the virtual selection region 625. Consequently, follicular unit 750 is harvested, as indicated in FIG. 7(f).
Returning now to the example we were discussing in FIG. 6(b), it can be seen that the follicular unit 640 was too far in the vertical direction from the horizontal boundary 610, and it was therefore not a desirable follicular unit to harvest at this time. The system, as shown in FIG. 6(b) illustrates follicular unit 640 still on the patient's body surface, and follicular unit 635 being harvested. On reaching the fiducial F, as illustrated in FIG. 6(c), the image processor ascertains that the revised vertical boundary 610 has been reached, and provides the control signals necessary for the robotic arm to move in the direction of arrow 650, as shown in FIG. 6(d). If desired, the movement of the tool in the direction of arrow 650 will allow the harvesting tool to harvest the follicular unit 640 that was previously left un-harvested. In FIG. 6(e), it can be seen that the virtual selection region 625 is moved in the direction 650 along the row F-C until all desired follicular units in that row are harvested. When no desired follicular units remain to be harvested within the desired “harvest quadrilateral” along the current row, the tool is operated to move in an upwardly direction 660 and towards the right, in the direction of arrow 665, to harvest follicular units in the row B-E in a similar manner, as illustrated in FIG. 6(f). Since hair and fiducials are in the same frame, it can be computed to determine whether the current harvest row needs to be incremented: move to the new row when there are no remaining hairs in the current row within the “harvest quadrilateral” formed by lines a specified distance away from the rows of fiducials. In the example of the robotic application, as the robot moves an automated harvesting (or implanting) tool along a current row and approaches a corner along the edge or boundary, the robotic system reverses direction and starts searching along a row spaced a configurable distance toward the opposite edge.
19. The system of claim 1, wherein the system is configured to allow a user to override the selected new medical procedure site or the automatically instructed movement of the tool to the selected new medical procedure site.
20. A system for controlling a direction of travel of a tool relative to a body surface during a medical procedure, the system comprising:
21. The system of claim 20, wherein the set of instructions comprises instructions for changing the direction of travel when another distinctive fiducial is within a predetermined distance from the tool and wherein the changed direction of travel is substantially opposite or substantially orthogonal to the direction of travel.
22. The system of claim 20, wherein the set of instructions comprises instructions for defining a boundary and controlling the direction of travel of the tool within the boundary.
23. The system of claim 20, wherein the set of instructions comprises instructions for defining an exclusion zone and/or a reserved region within which it is not desirable to perform the procedure or the second procedure.
24. The system of claim 23, wherein the exclusion zone or the reserved region comprises any closed polygon-shaped feature and has a pre-determined size, and wherein the size of the exclusion zone or the reserved region may be automatically generated, adjusted by a user, or both.
25. The system of claim 20, wherein the set of instructions comprises instructions for directing movement of the tool based on one or more selection criterion or criteria.
26. The system of claim 21, wherein the system is configured to allow a user to override any automatically identified direction of travel or automatically changed direction of travel.
27. The system of claim 20, wherein the tool is a hair harvesting or hair transplantation tool.
28. The system of claim 20, wherein the tool comprises a needle, a scalpel, a punch, a blade, an ablation tool, forceps, a surgical instrument, a suturing device, a laser, a cannula, a drill, a tattoo placement, or tattoo removal tool.
29. The system of claim 20, wherein the system is a robotic system and it further comprises a robotic arm and the tool is operatively connected to the robotic arm.
30. The system of claim 20, wherein the set of instructions comprises instructions for creating a visual representation of the first location and/or the second location after the procedure and/or the second procedure has been performed, and the system further comprises a monitor configured to display the visual representation.
US20150066054A1 true US20150066054A1 (en) 2015-03-05
US9743988B2 true US9743988B2 (en) 2017-08-29
WO2008109284A2 (en) 2007-03-07 2008-09-12 Ethicon Endo-Surgery, Inc Recognizing a real world fiducial in patient image data
KR20080049793A (en) 2005-09-30 2008-06-04 레스토레이션 로보틱스, 인코포레이티드 Automated systems and methods for harvesting and implanting follicular units
US20090005677A1 (en) 2007-06-19 2009-01-01 Adam Jerome Weber Fiducial localization
WO2010036513A1 (en) 2008-09-29 2010-04-01 Restoration Robotics, Inc. Tracking of hair follicles
Bouguet, "Pyramidal implementation of the Lucas Kanade feature tracker: Description of the algorithm", Intel Research Laboratory, Technical Report, 1999.2, 1999, 9 pages.
Communication Pursuant to Article 94(3) EPC in connection with commonly assigned European Patent Application No. 11851566.7, mailed Mar. 19, 2015. , (5 pages).
Communication under Rule 71(3) EPC, dated Nov. 20, 2015, in connection with commonly assigned European Patent Application No. 11851566.7 (10 Pages).
English Translation of Chinese Office Action mailed Jan. 6, 2015, in connection with commonly assigned Chinese Patent Application No. 201180080959.9 (7 pages).
English Translation of Japanese Office Action mailed Dec. 22, 2014, in connection in connection with commonly assigned Japanese Patent Application No. 2013-543418 (2 pages).
English Translation of Japanese Office Action mailed May 27, 2014, in connection, with commonly assigned Japanese Patent Application No. 2013-543418 , (2 pages).
English Translation of Korean Office Action mailed Dec. 24, 2014, in connection with commonly assigned Korean Patent Application No. 10-2013-7015821 (8 pages).
English Translation of Office Action mailed Aug. 18, 2015, in connection with commonly assigned Japanese Patent Application No. 2013-543418 (1 page).
English Translation of Office Action mailed Dec. 13, 2016, in connection with commonly assigned Japanese Patent Application No. 2015-245128 , (2 pages).
International Preliminary Report on Patentability, Forms PCT/IB/373 and PCT/ISA/237, issued Jun. 25, 2013., Jun. 25, 2013, 7 pages.
Jain, et al., "Machine Vision", 1995, Cover Page, Publication page, and Chapter 14, pp. 406-453, 50 pages.L.
Non-Final Office Action mailed Nov. 5, 2015, in connection with commonly assigned U.S. Appl. No. 13/796,159 , (12 pages).
Office Action mailed Jun. 22, 2014, in connection with commonly assigned, Australian Patent Application No. 2011349503 , (3 pages).
Office Action mailed May 27, 2014, in connection with commonly assigned, Canadian Patent Application No. 2,818,602 , (3 pages).
PCT International Search Report and Written Opinion for, PCT/US2014/022739 of the International Search Authority, Applicant Restoration Robotics, Forms PCT/ISA/210, 220 and 237, mailed Aug. 14, 2014 , ( 17 pages).
PCT International Search Report in relation to commonly assigned, PCT application PCT/US2011/065832, Form PCT/ISA/210, dated Aug. 31, 2012 , 7 pages.
Reply mailed Apr. 30, 2014 in response to Rule 70(2) and 70a(2) EPC Communication, (dated Nov. 12, 2013), in connection with commonly assigned European Patent Application No. 11851566.7 , 24 pages.
Sun, et al., "Real-Time Digital Image Stabilization Algorithm on PC", Proc SPIE vol. 4925. pp. 510-513. Sep. 2002 (Abstract in English), Sep. 2002, 2 pages.
Supplemental European Search in relation to commonly assigned, European Patent Application No. 11851566.7, Restoration Robotics, Inc. , (7 pages).