Patent ID: 12186043

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

The exemplary embodiments disclosed herein are based on an objective of providing a controller interface applicable to interchangeable manual and robotic control for an articulated steerable medical device having manual insertion mode and robotically controlled navigation mode to guide interventional tools and instruments, such as endoscopes and catheters, through intraluminal tortuous paths.

Throughout the figures, where possible, the same reference numerals and characters, unless otherwise stated, are used to denote like features, elements, components or portions of the illustrated embodiments. In addition, while the subject disclosure is described in detail with reference to the enclosed figures, it is done so in connection with illustrative exemplary embodiments. It is intended that changes and modifications can be made to the described exemplary embodiments without departing from the true scope and spirit of the subject disclosure as defined by the appended claims. Although the drawings represent some possible configurations and approaches, the drawings are not necessarily to scale and certain features may be exaggerated, removed, or partially sectioned to better illustrate and explain certain aspects of the present disclosure. The descriptions set forth herein are not intended to be exhaustive or otherwise limit or restrict the claims to the precise forms and configurations shown in the drawings and disclosed in the following detailed description.

When a feature or element is herein referred to as being “on” another feature or element, it can be directly on the other feature or element or intervening features and/or elements may also be present. In contrast, when a feature or element is referred to as being “directly on” another feature or element, there are no intervening features or elements present. It will also be understood that, when a feature or element is referred to as being “connected”, “attached”, “coupled” or the like to another feature or element, it can be directly connected, attached or coupled to the other feature or element or intervening features or elements may be present. In contrast, when a feature or element is referred to as being “directly connected”, “directly attached” or “directly coupled” to another feature or element, there are no intervening features or elements present. Although described or shown with respect to one embodiment, the features and elements so described or shown in one embodiment can apply to other embodiments. It will also be appreciated by those of skill in the art that references to a structure or feature that is disposed “adjacent” to another feature may have portions that overlap or underlie the adjacent feature.

The terms first, second, third, etc. may be used herein to describe various elements, components, regions, parts and/or sections. It should be understood that these elements, components, regions, parts and/or sections are not limited by these terms of designation. These terms of designation have been used only to distinguish one element, component, region, part, or section from another region, part, or section. Thus, a first element, component, region, part, or section discussed below could be termed a second element, component, region, part, or section merely for purposes of distinction but without limitation and without departing from structural or functional meaning.

As used herein, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should be further understood that the terms “includes” and/or “including”, “comprises” and/or “comprising”, “consists” and/or “consisting” when used in the present specification and claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof not explicitly stated. Further, in the present disclosure, the transitional phrase “consisting of” excludes any element, step, or component not specified in the claim. It is further noted that some claims or some features of a claim may be drafted to exclude any optional element; such claims may use exclusive terminology as “solely,” “only” and the like in connection with the recitation of claim elements, or it may use of a “negative” limitation.

The term “about” or “approximately” as used herein means, for example, within 10%, within 5%, or less. In some embodiments, the term “about” may mean within measurement error. In this regard, where described or claimed, all numbers may be read as if prefaced by the word “about” or “approximately,” even if the term does not expressly appear. The phrase “about” or “approximately” may be used when describing magnitude and/or position to indicate that the value and/or position described is within a reasonable expected range of values and/or positions. For example, a numeric value may have a value that is +/−0.1% of the stated value (or range of values), +/−1% of the stated value (or range of values), +/−2% of the stated value (or range of values), +/−5% of the stated value (or range of values), +/−10% of the stated value (or range of values), etc. Any numerical range, if recited herein, is intended to include all sub-ranges subsumed therein. As used herein, the term “substantially” is meant to allow for deviations from the descriptor that do not negatively affect the intended purpose. For example, deviations that are from limitations in measurements, differences within manufacture tolerance, or variations of less than 5% can be considered within the scope of substantially the same. The specified descriptor can be an absolute value (e.g. substantially spherical, substantially perpendicular, substantially concentric, etc.) or a relative term (e.g. substantially similar, substantially the same, etc.).

The present disclosure generally relates to medical devices, and it exemplifies embodiments of a catheter and/or an optical probe which may be applicable to an imaging apparatus (e.g., an endoscope). The imaging apparatus may image using a miniature camera based on chip-on-tip (COT) technology, or may provide some other form of imaging such as spectrally encoded endoscopy (SEE) imaging technology (see, e.g., U.S. Pat. Nos. 10,288,868 and 10,261,223). In some embodiments, the imaging apparatus may include an optical coherence tomographic (OCT) apparatus, a spectroscopy apparatus, or a combination of such apparatuses (e.g., a multi-modality imaging probe).

The embodiments of the optical probe and portions thereof are described in terms of their position/orientation in a three-dimensional space. As used herein, the term “position” refers to the location of an object or a portion of an object in the three-dimensional space (e.g., three degrees of translational freedom along Cartesian X, Y, Z coordinates); the term “orientation” refers to the rotational placement of an object or a portion of an object (three degrees of rotational freedom—e.g., roll, pitch, and yaw); the term “posture” refers to the position of an object or a portion of an object in at least one degree of translational freedom and to the orientation of that object or portion of object in at least one degree of rotational freedom (up to a total six degrees of freedom); the term “shape” refers to a set of posture, positions, and/or orientations measured along the elongated body of the object. As it is known in the field of medical devices, the terms “proximal” and “distal” are used with reference to the manipulation of an end of an instrument extending from the user to a surgical or diagnostic site. In this regard, the term “proximal” refers to the portion of the instrument closer to the user, and the term “distal” refers to the portion of the instrument further away from the user and closer to a surgical or diagnostic site.

As used herein the term “catheter” generally refers to a flexible and thin tubular instrument made of medical grade material designed to be inserted through a narrow opening into a bodily lumen (e.g., a vessel) to perform a broad range of medical functions. The catheter may be solely an imaging apparatus or it may comprise tools for use in therapeutic or diagnostic procedures. The more specific term “optical catheter” refers to a medical instrument comprising an elongated bundle of one or more flexible light conducting fibers disposed inside a protective sheath made of medical grade material and having an optical imaging function. A particular example of an optical catheter is fiber optic catheter which comprises a sheath, a coil, a protector and an optical probe. In some applications a catheter may include a “guide catheter” which functions similarly to a sheath.

As used herein the term “endoscope” refers to a rigid or flexible medical instrument which uses light guided by an optical probe to look inside a body cavity or organ. A medical procedure, in which an endoscope is inserted through a natural opening, is called an endoscopy. Specialized endoscopes are generally named for how or where the endoscope is intended to be used, such as the bronchoscope (mouth), sigmoidoscope (rectum), cystoscope (bladder), nephroscope (kidney), bronchoscope (bronchi), laryngoscope (larynx), otoscope (ear), arthroscope (joint), laparoscope (abdomen), and gastrointestinal endoscopes.

In the present disclosure, the terms “optical fiber”, “fiber optic”, or simply “fiber” refers to an elongated, flexible, light conducting conduit capable of conducting light from one end to another end due to the effect known as total internal reflection. The terms “light guiding component” or “waveguide” may also refer to, or may have the functionality of, an optical fiber. The term “fiber” may refer to one or more light conducting fibers. An optical fiber has a generally transparent, homogenous core, through which the light is guided, and the core is surrounded by a homogenous cladding. The refraction index of the core is larger than the refraction index of the cladding. Depending on design choice some fibers can have multiple claddings surrounding the core.

First, structural components of a robotic endoscope system1000comprising a robotic endoscope connectable to a handheld portable display controller500, and attachable to a support platform600will be described with reference toFIG.1A,FIG.1B, andFIG.2. The robotic endoscope can be continuum or multi-segment robot configured to form a continuously curved geometry by actuating at least a portion of a steerable endoscopic probe. An example of a continuum robot is a snake-like endoscopic device, as described in applicant's previously published U.S. Pat. No. 9,144,370, and patent application publications US 2015/0088161, US 2018/0243900, US 2018/0311006 and US 2019/0015978, which are incorporated by reference herein for all purposes.

<Configuration of Robotic Endoscope Probe>

FIG.1Aillustrates a general structure of an endoscope probe100controlled by a robotic control system300which is in data communication with a computer system400and/or a handheld portable display controller500, according to an embodiment of the present disclosure. The endoscope probe100has a non-steerable proximal section102and a steerable distal section101arranged in this order along a longitudinal axis Ax. The steerable distal section101includes a plurality of bending segments (1, 2, 3 . . . N); and at least one of those bending segments can be actuated to bend the endoscope probe100at an angle β with respect to the longitudinal axis Ax. The endoscope probe100is controlled by the robotic control system300which is mechanically connected to the probe100via a handle200.

The control system300generally includes a kinematic controller320, such as a proportional-integral-derivative (PID) controller or other digital signal processor (DSP) along with suitable software, firmware and peripheral hardware, which are known to persons having ordinary skill in the art. DSPs are generally dedicated integrated circuits; however DSP functionality can also be implemented by using field-programmable gate array chips (FPGAs). The control system300can be part of, or is connected to, a computer system400(e.g., a system console) which is operated by a processor or central processing unit (CPU)410. The computer system400, the robotic control system300, and the handle200are operably connected to each other by a network connection or cable bundle425. Alternatively or in addition, the control system300and/or handle200can be connected to the handheld portable display controller500via wireless or wired data communication, in a known manner. For example, the control system300or the handle200may include a wireless transceiver308to establish data communication with the portable display controller500via a corresponding wireless transceiver508.

Among other functions, the computer system400can provide a surgeon or other user with an image display device420and a graphical user interface (GUI)422to interact and remotely operate the endoscope probe100. Similarly, the handheld portable display controller500can provide the user with a control section502and an image display section505to interact and operate the endoscope probe100. The control section502may include manual controls such as a joystick503and a plurality of directional keypads504. The display section505may include a display screen for displaying a live image and/or a touch screen with a GUI. As discussed more in detail below, the portable display controller500is configured to be connected to the endoscope handle200for data communication, by direct attachment, by wire connection, and/or wireless connection therebetween. To that end, the portable display controller500may also include a data connection terminal507(e.g., RS323 connector, a USB or HDMI port) and/or wireless transceiver508(e.g., a WiFi® or Bluetooth®) transceiver.

The robotic control system300includes an actuator system310in data communication with the controller320. The actuator system310includes a plurality of actuators or actuating motors (Motor1through M) equal to a plurality of control wires110necessary for actuating and steering the probe100. The robotic control system300also includes and/or controls one or more sensors304. Sensors304can include a strain sensor and/or a position sensor which serve to detect and/or measure compressive or tensile forces applied by the actuators to drive each control wire110. These sensors304also output a signal305corresponding to the amount of compressive or tensile force (an amount of strain) being applied to a control wire110at any given point in time. The sensors304could also output a signal305corresponding to an amount of movement (a distance) of displacement for each actuated control wire110at any given point in time. The signals305from the sensors304(strain sensor and/or position sensor) for each control wire110are fed into the controller320to control each actuator and wire110individually with a feedback control loop. In this manner, each control wire110can be actively controlled to implement appropriate shaft guidance for navigating the probe100through intraluminal paths of a patient's anatomy.

The handle200provides electromechanical interface between the endoscope probe100and the control system300. For example, the handle200may provide mechanical, electrical, and/or optical connections, and a data/digital acquisition (DAQ) system for interfacing the endoscope probe100with the control system300. The handle200may also provide an access port250to insert medical tools, one or more mechanical dials or knobs252that an operator can use to manually control operations of end effectors and/or steering of the probe100, and a user interface254having one or more control buttons and status indicators. The handle200is attachable to a robotic platform600which includes a linear stage601to move the endoscope probe100in a linear direction L. The controller system300sends control signals to the platform600and linear stage601via the handle200or an additional connection such as a cable bundle615.

As part of the user interface254, the handle200may include one or more than one light emitting diode (LED) for providing operational status of the robotic endoscope probe100to a user. In an embodiment, the LED may include, for example, different light colors for respectively indicating normal operations (green light) and abnormal operations (red light). Alternatively, the LED may include blinking codes, for example, to indicate a type of abnormal operation. In addition, the user interface254may include an emergency on/off switch to manually stop actuation of the endoscope probe100, in the event of an emergency.

FIG.1Bshows a perspective view of an exemplary endoscope probe100. According to one embodiment, the endoscope probe100includes a sleeve or sheath made of a tubular non-steerable proximal section102and a steerable distal section101made of plurality of ring-shaped wire-guiding members arranged at a predetermined distance from each other. The endoscope probe100may have an outer diameter of about 0.14 inches, with the distal section101being around 2.0 inches in length, and the total length of the probe100being about 24 inches. In the distal section101, control wires110are anchored at one or more bending points (inflection points107) to ring-shaped anchor members. In the proximal section102, the control wires advance through the proximal end of the probe to the handle200. The wire-guiding members and anchor members are typically constructed from polyether block amide (e.g., Pebax®). A thin outer sheath and/or inner sheath made of medical grade lubricious material covers the outer and/or inner surface of the tubular shaft. The outer sheath can also be made of Pebax. However, Pebax is only one example, other composites or plastics are viable, e.g., polyurethane.

FIG.2illustrates in more detail the structure of endoscope probe100which includes an elongated flexible shaft (elongate body) commonly referred to as a sleeve or sheath (shown inFIG.1B). This sleeve or sheath has a tubular body having a proximal section and distal section, which includes one or more than one tool channel105and a plurality of wire conduits104spanning along the length of the sheath from the proximal end to the distal end. The tool channel105extends along (typically inside) a cylindrical opening of the tubular body (i.e., in the opening of the sheath), and the plurality of wire conduits104extend along (typically within) the wall of the sheath. As it can be appreciated by persons skilled in the art, tool channel105and wire conduits104are not limited to any specific cross-sectional shape; the cross-section of the tool channel and wire conduits can be circular, substantially circular, or polygonal as long as the functionality of the endoscope probe is not negatively affected.

The tool channel105includes an imaging device180, but it also allows passage for end effectors to be delivered and operated at the distal end of the sheath. For this reason, the imaging device180can be removably arranged in the tool channel105, so that the imaging device180can be repositioned within the tool channel and/or swapped for other components like end effector tools used during a procedure. In addition, tool channel105may also be used for delivering and/or retrieving fluids, such as liquid or gaseous substances (e.g., air, water) to/from a target area within a bodily lumen. The imaging device180may include an endoscope camera, or a fiber-based imaging probe. An example of an endoscope camera includes, but is not limited to, a chip-on-tip (COT) camera, such as a camera with a miniature CMOS sensor arranged at the tip of the imaging device. Examples of fiber-based imaging probes include, but are not limited to, a near infrared auto-fluorescence (NIRAF) imaging probe, a spectrally encoded endoscopy (SEE) probe, an optical coherence tomography (OCT) imaging probe, an intravascular ultrasound (IVUS) probe, or combinations thereof.

The wire conduits104allow anchorage and/or passage of control and support wires used for steering (bending, twisting or rotating) the steerable distal section101of the sheath. Some wire conduits104may be used for passing optical fibers and/or electrical cables. The bending segments (1, 2, 3 . . . N shown inFIG.1A) are joined to each other at one or more than one inflection point107. At the proximal end of the sheath, the handle200provides a physical interface between endoscope probe100and the control system300.

The endoscope probe100is configured to provide flexible access for an array of instruments allowing remote imaging, manipulation, cutting, suturing, delivery and retrieval of fluids, etc., to intraluminal target areas inside a patient's anatomy. To that end, the endoscope probe100must provide flexibility for navigating through tortuous paths, while retaining torsional and longitudinal rigidity so that the user can control the instruments located at the distal end of the sheath, by remotely operating such instruments with the control system300. To provide steerable functionality, the endoscope probe100is equipped with one or more support wires111and a plurality of control wires110, which are arranged inside the wire conduits104along (typically inside) the wall of the sheath. Some of the control wires110are anchored at the distal end of the sheath (in the distal section101) using wire anchors114. Other control wires110can be anchored at certain distances from the distal end (e.g., at an inflection point107) using wire anchors113.

In one exemplary embodiment, an endoscope probe100can be steered by six wires including one pair of support wires111and two pairs of control wires110. A first pair of control wires110can be anchored by wire anchors113at inflection point107of the steerable distal section101of the sheath. A second pair of control wires110may be anchored by wire anchors114at the distal end of the steerable distal section101. Support wires111are anchored to a fixed (stationary) structure211at the proximal end and to support anchors115at the distal end (or at an inflection point) of the steerable section101. The support wires111act as “tendon” wires to restore (counter) a steering action exerted by the control wires110. In some embodiments, the support wires111can be actively controlled in the same manner as the control wires110. In this manner, the endoscope probe100can have the steerable section101with at least two (i.e., two or more) bending sections each controlled by pairs of antagonistic wires running through wire conduits104. It is understood that each bending section can be controlled by more than two antagonistic wires.

While the case of driving control wires110for steering one or two segments of the steerable section101has been described above, if control wires110of all bending segments are driven, the postures of each bending segment may be independently controlled in accordance with the driving amounts of the individual control wires110. Further, a mechanism that twists or rotates the wire-driven endoscope probe100around the longitudinal axis may be provided. In the case of providing rotation or twisting action to the endoscope probe100, a first segment may be bent in a desirable direction by driving only one control wire110and then rotating the entire sheath about its longitudinal axis can control torsional movement of the distal tip of the sheath. Additional details for driving the snake robot include the control methods for actuation, as described in applicant's previous patent application publications US 2015/0088161, US 2018/0243900, US 2018/0311006, and US 2019/0015978, which are incorporated by reference herein for all purposes.

At the proximal end of the endoscope probe100, the handle200is configured to provide a mechanical linkage and an electromechanical interface between the endoscope probe100and the control system300. In one embodiment, the handle200provides a plurality of electromechanical connections210(one connection for each of the control wires110) so that the actuator system310can mechanically operate each control wire110individually. In addition, the handle200may provide an anchoring or fixing structure211(a mechanical ground) for the support wires111.

In some embodiments, electrical cables112are also passed through wire conduits104. The electrical cables112can be connected, on one end (at the proximal end) to a power source terminal212provided at the handle200, and on the other end (at the distal end) to an electronic component, such as tracking sensor, for example, an electromagnetic (EM) sensor190located in (typically inside) the wall of the sheath. For example, some endoscope probes may use the EM sensor190located at the tip of the instrument to obtain an electromagnetic signal (EM data) for sensing the position and/or orientation of the instrument's tip and controlling registration and navigation of the instrument, based on such EM data.

More specifically, as mentioned above, the electronic controller320is used to drive the actuators (motors) of the actuator system310to electronically control the operation of each control wire110by applying tension, torsional, or compressive forces to each control wire110. The controller320can control each control wire110by actively driving by an actuator or motor (310), a sensor (304or190), and a feedback control loop325to implement appropriate shaft guidance for navigating through tortuous intraluminal paths of a patient's anatomy. A typical 6DOF EM sensor with a sub-millimeter diameter and about 5 mm length can measure both position and orientation of the object to which such sensor is attached. Therefore, the EM sensor190can be used to measure the bending, twist, or rotation of the distal section101. In addition, the EM sensor190can track the position and/or orientation of imaging device180to maintain image orientation in the display section of the portable display controller500. Therefore, the signals305can include not only strain and displacement signals, but also position and/or orientation of the distal end of the steerable distal section to accurately track any twist, bend, or rotation of the probe.

In addition, either during insertion or retraction of the endoscope probe, the control system300controls the linear stage601of support platform600to move the probe100along the center line of the lumen (e.g., an airway) in a desired trajectory followed by active control of the bending sections. This is similar to known shaft guidance techniques used to control robotic guided catheters or endoscopes with the goal of forcing the flexible shaft to keep to the desired trajectory. In one example, when using a shaft guidance system, the endoscope probe is robotically advanced through a lumen while sensors measure the insertion depth of the shaft-guide and the angulations of user-controlled steerable tip segments to obtain trajectory information. The trajectory information is stored in a memory of the system and continuously updated. After a short advance in insertion or retraction distance, the shape of the steerable shaft-guide is corrected by adjusting (rotating or bending) segments of the instrument in such a way that the new shape closely matches the desired trajectory. This process is repeated until a target area is reached. A similar process can be applied when the endoscope probe is withdrawn from the patient. See, e.g., US 2007/0135803, which is incorporated by reference herein for all purposes.

However, before performing robotic controlled navigation along the length of the lumen, it is desirable to manually insert the probe100to a predetermined location (a first location) along the longitudinal direction of the lumen. For example, in robot-assisted bronchoscopy, it is advantageous to manually insert a steerable catheter to the first carina of the patient lungs, and then perform robot-assisted navigation thereafter (in the manner described above). Since, the probe can contact the patient's anatomy and this can cause discomfort and/or pain to the patient, the present disclosure provides a novel control mechanism with integrated display and control functionality that can be seamlessly operated in manual mode or robot mode.

Manual vs. Robot Mode—The Snake robot system includes a console with support platform (arm) and removable tethered handle. Disposable catheter is connected to handle and inserted into the patient, steered with manual mode to first carina. Manual mode has only simple functions like up/down/left/right that exist on current endoscopes. Once the catheter is at the first carina, the user attaches the handle to the support platform (arm), and the mode switches to robot mode. Once the system is in robot mode, the system controller operates the linear stage of the support platform, and the user can steer the probe in sophisticated robotically-enabled ways like the follow-the-leader or “chicken head” techniques. See, for example, U.S. Pat. Pub. 2018/0192854 and WO/2020/092097. These advanced movements are not possible without robotic control, but are enabled by portable display controller disclosed herein.

<Controller with Detachable Portable Monitor>

FIG.3Ashows an embodiment of an endoscope controller interface which is implemented by a handheld portable display controller500configured to be connected or attached to the endoscope probe100according a manual mode or robot mode of an endoscope procedure. The controller interface includes a detachable handheld portable display controller500which is connectable to an endoscope handle200, which in turn is connectable to a steerable endoscope probe100.FIG.3Bshows the endoscope handle200mounted onto a robot support platform600and wirelessly connected to the handheld portable display controller500. In manual mode, the user physically holds the display controller and the handle to manually insert and/or retract the probe into an anatomy of the patient while controlling the tip of the probe with the user interface (control section502). In robot mode, the motion of insertion or retraction of the probe is completely actuated by robotic control system300which controls the operation of the linear stage601and one or more links of the support platform600.

The handheld portable display controller500is a portable monitor which includes as least a control section502and a display section505. The portable display controller500generally includes, as part of the display section505, a display screen (e.g. an LCD or OLED display), a housing, illumination and imaging control electronics, image processing electronics, a power supply cable and/or a rechargeable battery. The endoscope handle200includes a connection indicator202, an access port250, and a cable bundle220. The connection indicator202is configured to inform the user that the handheld portable display controller500is operatively connected to the handle200. The connection indicator202may include, for example, a light emitted diode (LED), haptic feedback unit (vibrating surface), a sound emitting unit (speaker or buzzer), other similar device capable of informing the user of a connection status between the handle200and the handheld portable display controller500. The access port250is used to insert endoscopic tools into the tool channel105of the probe100. The cable bundle220includes at least a power cable and a video cable, and it serves to establish data communication between the endoscope handle200and computer system400.

As shown inFIG.3A, the handheld portable display controller500can be removably attached (physically connected) to the proximal end of handle200. However, as further describe below, the handheld portable display controller500can also be connected to the endoscope handle200by a wireless or wired network. When the handheld portable display controller500is connected to the handle200, at least part of the control section502is directly accessible to the user for manual operation while, at the same time, the user can observe an endoscope image on a screen of the display section505. For example, certain controls, such as a joystick, a multi-directional touchpad or keypad, or a touch screen GUI can be accessible to the user. The intent of this configuration is for manual insertion of the endoscope probe into a patient's anatomy prior to robotic control, so a simple joystick or directional keypad would suffice the required functionality to control the tip of the probe (catheter or endoscope). Other controls, such as buttons for activating irrigation or suction, can be disabled, eliminating the possibility of inadvertent activation and/or confusion by the user. Accordingly, the user can freely control the manual insertion of the probe100into a luminal anatomy of the patient without worrying about accidentally touching other buttons of control section502. Here, during manual operation, the probe movement in or out of the patient's anatomy can be user controlled, by manually pushing or pulling the handle200toward or away from the patient, while observing an endoscope image acquired by the probe and displayed on the handheld portable display controller500.

FIG.3Bshows the handheld portable display controller500mounted onto a robot support platform600. In this configuration, the handheld portable display controller500is connected to the handle200via a wireless connection510, but it is not physically attached to the proximal end of the handle200. In this configuration, while the endoscope handle200is mounted onto the support platform, the user can grab the handheld portable display controller500and have access to the control section502(e.g., joystick), for example, by using its thumb, while observing a live image in the display section505.

FIG.3Cillustrates a configuration where the endoscope handle200is mounted onto the robot support platform600and connected to the portable display controller500via wired connection512. In this manner, the handle200can establish data communication with both the portable display controller500(via the wired connection512) and with the main control system300(via the cable bundle220). The wired connection512may include any suitable video data interface (e.g., an HDMI interface) and/or a separate data connection (e.g., a USB connection). The structure of the support platform600is the same inFIG.3AandFIG.3B. In this embodiment, the cord of the wired connection512may be detachable from both ends, or it may be retractable on at least one end. In this manner, when the portable display controller500is not in use, the cord of connection512can be retracted into the handle200, into the controller500, or completely removed. In robot mode, the handle200is attached to the support platform600, and the corded portable display controller500is detached from the handle200and used for observing live images of the ongoing procedure. In some embodiments, the portable display controller500can be provided with a handle509to allow a secure grip by a user's hand. In this case, the control section502is strategically located adjacent to the handle509so that the user can control the joystick and/or buttons with a single thumb action. Moreover, the portable display controller500with a handle509can be configured with an auto-rotatable screen so that the user can use the controller500with either hand (i.e., with left-handed or right-handed control).

Once the endoscope handle200is docked on the support platform600, the control system300enters the robotic control mode. Specifically, as shown inFIG.3C, the portable display controller500is removed from the endoscope handle200, and the endoscope handle200can then be connected to the robotic control system300via the console cable bundle220. To facilitate safe removal of the portable display controller500, the endoscope handle200can be configured to inform the user of the attachment status thereof by activating or deactivating the connection indicator202when the manual operation is completed and the handle200is placed on the support platform600. In this manner, the user can be made aware that the portable display controller500is available for removal from the handle200.

FIG.4AandFIG.4Billustrate another embodiment of the portable display controller500removably connected to the endoscope handle200. The concept described in this embodiment is the portable display controller500implemented as a removable, tiltable handheld monitor with a simple joystick control for maneuvering the catheter tip during manual insertion. In this embodiment, the endoscope handle200is provided with a proximal connecting portion208which is configured to interface with a similar connection in the portable display controller500. The connection portion208can be, for example, a RS232 or similar type connector.

In the embodiment shown inFIG.4AandFIG.4B, the user first grips the handle200to control the probe (catheter insertion/retraction) movement. At the same time, while the user is inserting (or retracting) the probe100, the user is monitoring the camera view on the monitor (display section505), while manually steering the tip of the probe100using the joystick503and/or directional keypads504of the control section502.

Once a predetermined location within the patient's anatomy is reached (e.g., once the first carina of the lungs is reached), the portable display controller500is removed (detached) from the handle200, and the handle200is attached to the support platform600to initiate navigation and steering under robot mode. In the robot mode, the user now uses the portable display controller500(portable monitor) and the control section502(joystick and/or keypads) to control the snake-like movement (navigation) of the robotic probe for the remainder of the procedure.

Some of the main elements in the foregoing embodiments of a robotic endoscope system1000include an endoscope handle200combined with a tilting, detachable handheld monitor (portable display controller500) to perform a manual control mode for part of an endoscopic procedure. The handheld monitor may included a liquid crystal display (LCD) or an organic light emitting diode (OLED) display device of reduced size and weight substantially similar to a “smart phone” or a portable “video game console”; this handheld monitor may or may not have touch screen capability. The handheld monitor may have a joystick and/or a directional keypad to control the catheter tip during manual insertion mode. The joystick and/or directional keypad (e.g., a touchpad) is a control section which may be incorporated in the handheld monitor, or it can be provided as a separate device to be removably integrated with the handheld monitor.

Once the handheld monitor is detached, the handle200is attached to a support platform that is possibly connected to the patient bed, the system cart (console), or other secure site. In any configuration, attaching the handle200to the support platform initiates robot mode. When in robot mode, the insertion/retraction of the probe is accomplished by the support platform mechanically moving the handle200and the probe in a linear direction (e.g., up/down or forward/backward along the length of the support platform). This movement might be accomplished through known electromechanical controls using typical metal toothed gears, drive belts, magnets, telescopic links, or combinations thereof. Robotic controls of this nature are known to persons skilled in the art from, for example, U.S. Pat. Nos. 10,188,471 or 8,682,416, which are incorporated by reference.

<Robot Support Platform for Handheld Portable Display Controller>

FIG.5,FIG.6, andFIG.7disclose further details of the support platform600and the portable display controller500which are part of the robotic endoscope system moo. The support platform600includes a support plate602, a linear stage601, and one or more than one robotic arms. The support platform600is configured to attach the snake robot system1000to a patient bed or to a console cart unit. The support plate602, which is a place for the user to attach the handle200, may also be configured to store, and/or recharge the portable display controller500(a handheld monitor). In these exemplary drawings, the portable display controller500includes the display section505and the control section502separable from each other. In these examples, the display controller500includes various components similar to those of, for example, the Nintendo Switch console, available from Nintendo Co., Ltd. (Kyoto, Japan).

InFIG.5, the support platform600includes a robotic arm610, a support plate602, a linear stage601, a monitor receptacle605, and one or more joints609. The robotic arm610may comprise a multi joint arm that includes a plurality of linkages connected by joints609(having non-illustrated actuators and encoders) to enable the linkages to rotate, bend and/or translate relative to one another in response to controlling signals from the robot control system300. The robotic arm610may be fixed to a support structure (patient bed or system console) at one end and may have an end effector linkage at the other end with various degrees of freedom.

The support plate602is an attachment feature for holding the handle200of the endoscope probe100. The linear stage601is a controllable feature configured to physically engage with handle200and actuate linear movement the probe100in small increments of movement. The monitor receptacle605is a place for the user to attach, stow, view, and/or recharge the portable display controller500. In operation, under the manual mode, the handle200is removed from the support plate602and connected to the portable display controller500; the user manually inserts the probe100into a lumen while observing live view images in the display section505and steering the probe tip with the control section502of the controller500. Upon reaching a predetermined length of insertion, or reaching a predetermined anatomy, the user disconnects the portable display controller500, and returns the handle200to the support plate602. Once the endoscope handle200is attached to the support platform, the system enters a robotically controlled mode. Under the robotically controlled mode, control system300controls the support platform to insert and/or retract the endoscope probe100into the patient by linear movement of the handle200along the linear stage601.

Optionally, the user, via the control section502and the display section505of the portable display controller500, can control and/or observe the actions of the tip of the endoscope probe100. Specifically, the portable display controller500can be used as a remote control for controlling the bending of the endoscope probe tip and possibly other functional controls such as irrigation/suction or image capture. The portable display controller500can be a wireless controller which can be docked onto the monitor receptacle605of the platform support600. Providing the monitor receptacle605in the robotic arm610, for example, serves at least three functions, including: (1) provides a place to store the portable display device, (2) provides a place to recharge the wireless portable display device or provides a place to stow the wired portable display device when not in use, and (3) provides a place for the user to temporarily park the portable display device during parts of a workflow procedure that requires the user to have both hands free.

FIG.6illustrates a further alternate embodiment of the support platform600having a hidden monitor receptacle605. In this embodiment shown inFIG.6, the portable display controller500can remain hidden behind the support plate602when not in use, and is moved in the directions of arrow506when in use. In this case, the user can pull the portable display controller500out from the monitor receptacle605for viewing live images of a procedure. In addition, the user also has the option to completely detach the portable display controller500from the support platform600for handheld (manual) control and image viewing. Once a procedure is completed, the user replaces the monitor back in the monitor receptacle605behind the support plate602. This configuration allows for a compact, yet versatile environment where the user can advantageously use the portable display controller500for manual controlled insertion and robotic controlled navigation while continuously observing the procedure. When the portable display controller500is not in use, it is returned to the monitor receptacle605, which can serve as a display charging station. In addition, while the display controller500is in the monitor receptacle605, the display is protected from damage during transport and/or movement in crowded spaces such as a procedure room. Making the portable display controller500removable and storing it in a hidden monitor receptacle605is important given how frail display screens are, and given that a procedure room likely contains video monitor equipment, workstations, fluoroscopy equipment, Pyxis medication dispensing stations, a patient bed, an anesthesia cart, and various storage cabinets.

<Console and Control System with Handheld Portable Display Controller>

FIG.7Aillustrates a robotic endoscope system1000with an embodiment of the support platform600for controlling an endoscope probe100using the handheld portable display controller500in manual mode or robot mode. In this embodiment, the display section505of the portable display controller500(handheld monitor) is removably mounted in a monitor receptacle605which is attached to a repositionable support arm606. The control section502is separated from the display section505, and mounted on a side of the support plate602. This arrangement can be advantageous in order for the user to optimize viewing angles and ease of movement during the operation of the system.

Specifically, according toFIG.7A, the robotic endoscope system1000includes a console700comprised of a computer cart710and a main display device720. The computer cart710encloses therein the computer system400. The main display device720may include a screen722(corresponding to screen422described inFIG.1A). The support platform600is a multi-link robotic arm attached to the computer cart710of console700. The support platform600includes the support plate602, a linear stage601, a first arm link603, a second arm link604, a repositionable articulated arm606, and a monitor receptacle605. The monitor receptacle605can be provided with magnetic contacts or spring loaded snap-on brackets configured to hold therein the portable display controller500in a secure, but removable manner. The articulate arm606can have a plurality of secondary links with various degrees of freedom in order for the user to optimize viewing angles of the display section505(screen) during the operation of the system.

The robotic endoscope system1000is configured to perform an endoscopy procedure on a patient (P) which is disposed on a medical table or bed (B). To that end, similar to other embodiments, the user (not shown inFIG.7A) may use a control section502(e.g., a handheld joystick controller) to perform initial manual insertion of the endoscope probe100to a predetermined location of the patient's anatomy (e.g., the user may perform manual insertion of a bronchoscope to the first carina of a patient), while observing the manual insertion on the display section505(a display screen) of the portable display controller500. In addition, the portable display controller500and the control section502can be configured to be combined with each other and with the handle200, e.g., to form a single controller (as shown inFIG.3A), so that the user can use both the control section502and screen display section505for manual insertion as previously described. After the manual portion of the procedure is completed (after the manual mode), the user can return the control section502, the handle200, and the display section505of portable controller500to their respective positions on the support platform600. As soon as the endoscope handle200is returned to the support plate602, the control system300takes over the robotic control (enters robot mode) to robotically navigate the probe for the remainder of the procedure. In robot mode, the controller320kinematically controls the actuators or motors to bend, twist or rotate the probe via one or more of the control wires110.

In robot mode, the robotic control system300(or a separate control system) may also control the movement of one or more robotic arms or links of the support platform600. The robotic control system300may receive sensor data from each robotic arm or link thereof indicating current parameters of the robotic arm or link thereof (e.g., position, joint angles, measured forces, etc.) and may send control signals to actuators (gears, belts, magnets) to drive the movement of the arm610or links603,604, and606. A motion tracking system (not shown) may track the position and orientation of the robotic arm and links thereof to determine the position of the support plate602within the coordinate system of the patient. A control loop, which may be executed using the image-guided procedure of the endoscope probe100, the motion tracking system (not shown), and the robotic control system300, may continuously read position and orientation data of the endoscope probe100and the robot arm parameters data and may send controlling signals to the robotic control system300to cause the robot support platform600to move the support plate602, the linear stage601, robot arm610, and/or the monitor receptacle605to a desired position and orientation to optimize viewing angles and navigation control.

In this embodiment, display device720may be a primary display device (e.g., a main monitor) that may be connected to the endoscope handle200by a wired or wireless link (e.g., cable bundle220shown inFIG.3A-3C). In one embodiment, the system console700may receive live-view image data (video data) from the endoscope probe100, e.g., via the cable bundle220, and output the video data to the main display device720and to the portable display controller500over a suitable video data interface (e.g., an HDMI interface) and may also exchange other signals with the display device over a separate data connection (e.g., a USB connection). In addition or alternatively, the portable display controller500may the receive live-view image data (video data) from the endoscope probe100, e.g., via wired or wireless data communication.

FIG.7Billustrates functional blocks of computer system400(shown inFIG.1A) which may also operate or be part of the portable display controller500. As shown inFIG.7B, the computer system400may include, among other things, a central processing unit (CPU)410, a storage memory412including volatile random access memory (RAM) and non-volatile read only memory (ROM), a user input/output (I/O) interface413, and a system interface414which are operatively interconnected via a data bus415. The computer system400can be programmed to issue control commands which can be transmitted to the various parts of the control system300, the system controller320, the endoscope handle200, and/or the display controller500, e.g., upon receiving a user input via the user interface413. A touch panel screen, a key board, mouse, joystick, ball controller, and/or foot pedal can be included as part of the user interface413. For example, the display device420and GUI422shown inFIG.1Acan be part of the user interface413. Using the user interface413, the user can issue a command to cause the control system300to actively operate the steerable probe100. For example, when a user inputs a command via the user interface413, the command is transmitted to the central processing unit CPU410for execution of a given program routine thereby causing the CPU410to send a command via the system interface414to one or more of the motors or actuators in the actuator system310, the linear stage601, or to read signals output form one or more strain sensor304and/or EM sensor190.

The CPU410may include one or more microprocessors (processors) configured to read and perform computer-executable instructions stored in the storage memory412. The computer-executable instructions may include program code for the performance of the novel processes, methods and/or calculations disclosed herein. In particular, computer-executable instructions may include program code for executing the processes illustrated inFIG.8A,FIG.8B, andFIG.9to implement the real-time and seamless transition of manual mode and robot mode control of the endoscope probe100.

The storage memory412includes one or more computer readable and/or writable media, which may include, for example, a magnetic disc (e.g., a hard disk), an optical disc (e.g., a DVD, a Blu-ray), a magneto-optical disk, semiconductor memory (e.g., a non-volatile memory card, flash memory, a solid state drive, SRAM, DRAM), an EPROM, an EEPROM, etc. Storage memory412may store computer-readable data and/or computer-executable instructions.

The system interface414provides electronic communication interface to input and output devices. In particular, system interface414may include one or more electronic circuits, such as a field-programmable gate array (FPGA) circuit boards to interface the computer system400with the motors or actuators that operate the control wires110. The system interface414may also include interface connections for communication cables and a network port (either wired or wireless).

As it will be appreciated by a person of ordinary skill in the art, the location of the processor for the portable display control device is not necessarily limited to the described structures. A processor or computer, as illustrated inFIG.7B, could be located in the controller500itself, on the handle200, in the cart (as shown inFIG.7A), or maybe even located on a network such as when using cloud-based processing. Moreover, the portable display controller500can be configured to take advantage of augmented computing capabilities by proprietary artificial intelligence (AI) tools that can help identify regions of interest for actuation and navigation of the probe100based on endoscope imaging cases performed in referential subject all around the world. Furthermore, portable display controller500and/or robotic system can two or more processors and operating systems: one for the back-end and another for the front end, e.g., for the touch screen user interface. The processor for this user input device could be built into the device itself (like a Nintendo Switch) or the user input device could be simply a touch screen whose processor is somewhere else.

FIG.8AandFIG.8Billustrate a robotic endoscope system1000with a support platform600for controlling an endoscope probe100using the handheld portable display controller500in manual mode and robot mode, respectively, according to another embodiment. Design and usability of modern robotic endoscope systems strive to satisfy the user's needs, expectations, and limitations, at the same time as providing patient safety and fulfilling potential for market acceptance with a cost-effective solution. For example, it is not uncommon that a typical endoscopy procedure room would likely contain video monitor equipment, workstations, fluoroscopy equipment, automated medication dispensing stations, a patient bed, an anesthesia cart, and various storage cabinets, all interconnected by wiring and tubing connections. Therefore, new endoscope systems must accommodate to typical procedure workflows, constrained space limitations, and common procedural practices. In this regard, for example, it is important to satisfy recommend standards of button-size, foot-pedal location, and other human-robot interface parameters, such as weight, size, force requirements of a control interface to address the human factor needs of users and patients.

FIG.8AandFIG.8Billustrates a cart-mounted support platform embodiment, as an example of a proposed robotic endoscope system1000configured to satisfy most of a user's needs and patient's safety requirements. A cart-mounted support platform can advantageously provide compatibility with existing workflows. In addition, a small-sized handheld portable display controller (a mini-monitor) can provide additional functionality without taking up space in the crowded environment of a procedure room. According toFIG.8AandFIG.8B, a snake robot system configuration includes at least a cart-mounted support arm; and a bronchoscope-like or a gamepad type controller interface to operate a steerable endoscope probe100under an unattached manual insertion mode, and a robotic support platform attached robot mode.

Similar to previous embodiments, the robotic endoscope system1000includes a console800, a cart-mounted support platform600, a handle200attached to a proximal end of an endoscope probe100, and a handheld portable display controller500. A patient (P) disposed on a bed (B), in a supine position, is the subject of an endoscopic interventional procedure. The support platform600includes a first robot arm610and a second robot arm620. The console800includes a modality cart830and a display device820. The modality cart830includes the snake robot control unit (robot control system300shown inFIG.1A). The display device820has a main display screen822(corresponding to display screen422ofFIG.1A) configured to provide a user with a graphical user interface (GUI) for interacting with and controlling the endoscope probe100. The portable display controller500is normally docked on a monitor receptacle or clamp805. The first robot arm610includes one or more actuated arm links configured to hold and position the linear stage601aligned with respect to the patient P. The second robot arm620includes one or more actuated arm links configured to hold and position an electromagnetic field generator850. The EM field generator850is used in conjunction with the EM sensor190included in the catheter sheath to generate an EM tracking signal used for registration and/or navigation.

Under the manual mode, as shown inFIG.8A, a user (U), for example, an endoscopist removes the endoscope handle200from the support platform600, and attaches the portable display controller500to the proximal end of endoscope handle200. The user then uses the combined portable display controller500connected to the handle200and endoscope probe100to perform a manual intubation (manual insertion) procedure in which the probe100is manually inserted to a predetermined location through an anatomy of a patient (P). The user can use the control section502of the portable display controller500to control the tip of endoscope probe100during the manual insertion mode. After manual insertion, the user returns the endoscope handle200to the support platform600, and may also attach the portable display controller500to the display receptacle805on the robot arm610. Under the manual mode, the user grips the handle200and/or the portable display controller500to dictate probe insertion/retraction. While the user is inserting the probe, the user is monitoring the camera view on the monitor (display section505), while steering the probe tip using the catheter tip control. Once a predetermined anatomy is reached (e.g., once the first carina of the lungs is reached), the monitor/controller is detached, and the handle is returned to the support platform to initiate robot mode.

Under robot mode, as shown inFIG.8B, the user now uses the monitor of the portable display controller500to observe the endoscope image, and the joystick to control the probe100for the remainder of the procedure. More specifically, under robot control mode, the system300controls the linear stage601to move the handle200in a linear direction L, while the user can still observe the images in the main display822or the display section505of the controller500. In addition, during the robot mode, since the portable display controller500is attached to the robot arm610, the user can continue to actuate the tip of the endoscope probe100with a separate joystick controller860while performing an interventional procedure (such as ablation) with a tool inserted through access port250, as outlined in the process ofFIG.9below.

<Control Method for Handheld Portable Display Controller>

FIG.9shows an exemplary workflow which defines a process (method) for manual insertion and robotically controlled navigation using the handheld portable display controller500, according to the various embodiments described above. When in manual mode, the user manually inserts and retracts the steerable endoscope probe100, while the user controls the tip articulation by using actuators in the system with commands via a controller interface, such as a joystick or keypad. When in robot mode, besides the tip articulation, the user also uses the actuators to insert and retract the probe with commands via the controller interface. In both the manual mode and robot mode, the user has direct access to the display section505to observe, and optionally interact, with the endoscope images acquired by the probe.

According to the workflow ofFIG.9, at step S902, a patient preparation step occurs. For example, the user (an endoscopist) manually inserts an endotracheal tube into a patient (P). At step S904, the user attaches a new (sterile) catheter or probe100to the handle200which is already attached to the support platform600. The new catheter or probe100is generally provided in a straight packaging sleeve. In addition, at step S904, the handheld portable display controller500is now connected to the endoscope handle200. Next, at step S906, an auto-calibration process occurs. At step S906, because the catheter or probe100is provided in a straight packaging sleeve, the system can calibrate the positions of the control wires to a mean straight catheter position; this is an operation analogous to a “tare” process of a weighing scale, so that probe navigation can start from a known reference position and orientation. In this step, for example, in order to register the reference position and orientation of the probe, the system can control the imaging device180(camera) to initiate acquiring live-view images. Alternatively, the system can activate the EM tracking system to register the position and/or orientation of the EM sensor190with reference to the patient and/or bed coordinates.

At step S908, the user removes the straight packaging sleeve from the sterile catheter or probe100. In other embodiments, the sterility of the catheter may be maintained via other processes or steps, so the packaging sleeve may not be used. At step S910, the user removes the combined handle200and portable display controller500from the support plate602, and moves the assembled probe100and display controller500to the patient. At step S912, the user manually inserts the catheter or probe100to a predetermined location using manual steering controls (e.g. using the control section502), while observing the procedure on the screen display section505. For example, for a bronchoscopy procedure, the user manually inserts the catheter or probe100through the mouth of the patient (P) to the first carina, moving his arm in a linear direction L, while steering the tip of the probe with a joystick, control knob, or directional keypad of the control section502. In this step, the user can observe the endoscope live-view image in the display section of the portable display controller500. The processes of steps S910-S912can be considered as a first procedure of the catheter or probe100in which the connected portable display controller500and handle200allow the user to use the control section502to manually steer the distal section of the probe while observing an image of the procedure in the display section505of the portable display controller500. While this manual operation can be an almost linear insertion operation, there is still a need to steer the tip of the probe during the manual insertion to avoid or at least minimize patient discomfort. Here, the manual steering of the tip of the probe can be done via the joystick or keypad of control section502.

After manually positioning the distal end of the probe at the predetermined location (first location), at step S914, the user brings the support platform600(e.g., by moving an articulated arm) closer to the patient's location and attaches the handle200and controller500back onto the support plate602. Attaching the handle200onto the support platform600causes the system to enter robotic control mode. In some embodiments, attaching the handle200onto the support platform600may disable manual control mode. However, at step S916, the user can still use the portable display controller500and/or the main display device (820inFIG.8) of the console to confirm that the distal tip of the endoscope probe100has remained positioned at the desired initial location. Depending on the application, the process to confirm probe location can be done with an additional modality, for example, by endobronchial ultrasound (EBUS), or radial EBUS (REBUS), or by using EM data obtained from the EM generator850and EM sensor190.

In other words, step S916is a process to confirm (or register) the location of the probe's tip in the subject's anatomy (first carina location) with the software application of the robotic control system. After confirming the location of the probe's tip, the user can now detach the handheld portable display controller500from the handle200to continue probe navigation under robot controlled mode (step S918). In robot controlled mode, the robot control system300(robotic controller320) controls the linear stage601in the support platform600(robotic arm) to linearly move the handle200in a direction L to continue moving the probe along the lumen of the patient's anatomy, while continuing to steer the tip of the probe100.

At step S920, during robot controlled mode, the user can still use the joystick, keypad or knob controller of the control section502to navigate the tip of the probe100to the desired target lesion while continuing to acquire and observe the on-screen endoscopic images500on the display screen section505of the portable display controller500. Alternatively, the screen section505of the portable display controller500may be used as a display to navigate to the target lesion using one or more of ultra-sound navigation, EM navigation, fluoroscopy, or the like. The process of steps S914-S920can be considered as a second procedure of the probe100in which the portable display controller500remains in data communication with the handle200to allow the user to robotically control navigation (e.g., to remotely actuate the tip of the probe) inside the anatomy of the patient (e.g., using the control section502or using a touch screen GUI) while the robot control system300controls linear movement (insertion or retraction) of the probe.

In this manner, in the manual mode, the user manually inserts the probe tip towards a first location while controlling the tip of probe100using the handheld portable display controller500, and in the robot mode, the robot control system controls the linear movement of the probe for robotic navigation from the first location to a second (target) location while the user can still control the orientation of the probe tip (if necessary). As noted above, controlling the probe tip can be effected by actuating control wires of the endoscope probe by a joystick controller or by a touch screen GUI in the display section of the portable display controller. In particular, a touchscreen display operative to control angulation of the tip probe and to display operations of the probe without the need for using traditional keyboard controls can lead to more efficient procedures without leaving the patient's bedside or shifting attention from the patient to communicate with assistant personnel or observe remotely mounted displays. Therefore, the ability to use a combined handheld portable display and manual controller device that can be efficiently attached and detached from the endoscope handle is considered of significant advantage over conventional technology.

The foregoing disclosure describes various embodiments of a handheld portable display controller500configured to interact with a robot support platform of a robotic endoscope system for combined manual insertion and robotic controlled navigation of an endoscope probe100. The endoscope probe has a handle for the user to grip while manually inserting the catheter to a predetermined location (e.g., the first carina of the lungs). The handheld portable display controller500can be connected to the handle200for manual control. After manual insertions, the handle is attached to the support platform. This initiates robotic mode, where the insertion/retraction is controlled and moved by the support platform of the robotic system. When the system is in robotic mode, the handheld portable display controller can be detached from the handle and may be used to control the system and observe endoscope images of a procedure.

According to one embodiment, the handheld portable display controller500is a removable, tiltable monitor (a mini-monitor) with a joystick control for maneuvering the catheter tip during manual insertion. For manual insertion, the user grips the handle of the endoscope to control catheter insertion/retraction. While the user is inserting/retracting the catheter, the user is monitoring the camera view on the monitor, while steering the tip using the joystick. In an exemplary procedure, once the first carina of the lungs is reached, the monitor/controller is disconnected from the handle, and the handle is attached to the support platform to initiate robot control mode. Under robot mode, the user can use the portable display controller to control the navigation of the probe and observe live-view images for the remainder of the procedure.

Under manual mode, robotic system control is not available, but the portable display controller provides simple actuation controls such as up/down and left/right actuation of the probe to the user. Advantageously, the manual mode requires Low Cognitive load, as it provides basic functions without causing confusion for the physician. In addition, manual mode provides avoidance of use error—because it is so simple, it avoids use related errors that come from complicated user interfaces. Speed—It's fast and relatively easy for endopscopists to navigate to the first carina today with non-motorized endoscopes and their near-instantaneous responsiveness. SNAKE robot systems are motorized and rigidly mounted to complex robotic arm platforms, so it is much much slower than an endoscope operated by hand. By keeping the feature list small and similar to what endoscopists are used to, the controller makes insertion much faster than it would be if the user had to navigate to the first carina using robot mode. Ergonomics—The controller is a lightweight device in the physician's hand (not a giant endoscope that they have to hold perfectly still at a weird angle or lose their spot). Also, by having handheld monitor, the physician can focus on the patient and the procedure without looking across room.

Under robot mode, robotic control mode is fully enabled for linearly advancing or retracting the probe, and the portable display controller continues to provide simple controls such as up/down and left/right actuation, as well as advanced “follow-the-leader” or “chicken head” actuation of the probe to the user. In this manner, robot mode enables more advanced functionality. Specifically, the controller makes advanced robotic functions possible, allowing physicians to reach places in an intuitive and interactive manner which was not previously possible. For example, in a case where the user decides “I want to go there”, the user can touch the image on the screen where him/her wants the snake to go (e.g. pick a branch), and the probe tip goes there.

Software Related Disclosure

Embodiment(s) of the present disclosure can be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like. An I/O interface can be used to provide communication interfaces to input and output devices, which may include a keyboard, a display, a mouse, a touch screen, touchless interface (e.g., a gesture recognition device) a printing device, a light pen, an optical storage device, a scanner, a microphone, a camera, a drive, communication cable and a network (either wired or wireless).

Other Embodiments or Modifications

In referring to the description, specific details are set forth in order to provide a thorough understanding of the examples disclosed. In other instances, well-known methods, procedures, components and circuits have not been described in detail as not to unnecessarily lengthen the present disclosure. Unless defined otherwise herein, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The breadth of the present invention is not to be limited by the subject specification, but rather only by the plain meaning of the claim terms employed.

In describing example embodiments illustrated in the drawings, specific terminology is employed for the sake of clarity. However, the disclosure of this patent specification is not intended to be limited to the specific terminology so selected and it is to be understood that each specific element includes all technical equivalents that operate in a similar manner.

While the present disclosure has been described with reference to exemplary embodiments, it is to be understood that the present disclosure is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.