Abstract:
A system and method for fault detection and recovery in a medical imaging system are provided. The method includes monitoring operations of a plurality of subsystems of the medical imaging system and determining if an error has occurred in a subsystem based on the monitoring. The method further includes identifying at least one component within the subsystem having the determined error and automatically restoring the at least one component in the identified subsystem having the determined error.

Description:
BACKGROUND OF THE INVENTION 
     The invention relates generally to medical imaging systems, and more particularly, to a system and method for fault detection and recovery in such medical imaging systems. 
     Medical imaging systems, such as, for example, computer tomography (CT) systems, X-ray systems, magnetic resonance imaging (MRI) systems, positron emission tomography (PET) systems and the like, are defined by a number of subsystems that interact to deliver the functionalities for these systems. In an X-ray scanner, such subsystems include, but are not limited to an X-ray generator, a table positioner, a system control and an operator console. These subsystems are typically different physical computer systems running on different operating systems. 
     The subsystems may be further subdivided into software applications or other logical subsystems. These software applications are typically object oriented programs that individually or in combination with other software applications perform specific functionality. For example, the functionality may include image acquisition, image processing, etc. 
     The complexity in software applications and the interrelationship between applications has increased. This also has increased the criticality of faults and their occurrence. In medical imaging systems, testing and maintenance reduces the likelihood of these faults, however, faults and errors still occur. This then may result in X-Ray scanner downtime, thereby decreasing throughput (e.g., reducing the medical procedures that can be performed) and increasing costs. 
     Additionally, known techniques in fault management in medical imaging systems require rebooting the whole medical imaging system, even for a fault that is not hardware based or that only affects specific components within the system. This reduces the operating time of the medical imaging system, which again reduces throughput and increases costs. 
     BRIEF DESCRIPTION OF THE INVENTION 
     In one exemplary embodiment, a method for controlling a medical imaging system is provided. The method includes monitoring operations of a plurality of subsystems of the medical imaging system and determining if an error has occurred in a subsystem based on the monitoring. The method further includes identifying at least one component within the subsystem having the determined error and automatically restoring the at least one component in the identified subsystem having the determined error. 
     In another exemplary embodiment, a medical imaging system having a plurality of subsystems is provided. The subsystems each further include at least one component. The medical imaging system further includes a subsystem controller for each of the plurality of subsystems and communicating with the at least one component in each of the subsystems. The subsystem controller is configured to identify components within the subsystem failing to respond to a request for a response and to reset the component failing to respond. The medical imaging system additionally includes a system manager communicating with each of the plurality of subsystem controllers and is configured to communicate a failure to respond to each of the subsystem controllers. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is block diagram of a medical imaging system constructed in accordance with an exemplary embodiment of the invention. 
         FIG. 2  is a detailed block diagram showing the sub-elements of the medical imaging system of  FIG. 1 . 
         FIG. 3  is a block diagram of a subsystem controller in accordance with an exemplary embodiment of the invention that may be used in connection with the medical imaging system of  FIG. 1 . 
         FIG. 4  illustrates charts showing exemplary severity levels in accordance with an exemplary embodiment of the invention. 
         FIG. 5  is an exemplary configuration file in accordance with an exemplary embodiment for use in various embodiments of the invention to assign severity level values to actors. 
         FIGS. 6A and 6B  show a flowchart of a process for fault handling and recovery of a medical imaging system in accordance with an exemplary embodiment of the invention. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     Various embodiments of the invention provide fault management and recovery in a medical imaging system, including, for example, recovery due to software related faults. Software related faults may include, but are not limited to, a crash, hang up, lock-up, failure, error and/or other undesirable process that may lead to the malfunctioning of the software or medical imaging system. Further, the medical imaging system of the various embodiments, may include, for example, a Computer Tomography (CT) Scanner, Medical Resonance Imaging (MRI) Scanner, Positron Emission Tomography (PET) Scanner, Ultrasound Scanner, and/or an X-ray Machine. 
     In general, the various embodiments provide fault management and recovery performed based on different levels of severity of the fault. The different levels of severity of the various embodiments of the invention determine in part the interaction and operation of the hardware in the fault recovery process. As a result, the reboot of the medical imaging system is only required in some cases, for example, if the fault related to both software and hardware of the medical imaging system. In operation, the fault recovery mechanism of the various embodiments of the invention returns the component or the medical imaging system to the stable state, for example, in which it was operating before the fault. In general, the stable states of a medical imaging system include, but are not limited to, system idle, application setup, and exam-started state. The system idle state represents a state wherein no operations are being performed on a medical imaging system. The application setup state represents the condition wherein the radiologist/user selects the application protocols to perform on the patient. Further, the exam-started state represents a state wherein a medical exam is being performed on a medical imaging system. 
       FIG. 1  shows a medical imaging system  100  that performs fault management and recovery in accordance with various embodiments of the invention. Medical imaging system  100  in its various embodiments includes one or more physical subsystem, for example, a physical subsystem  102  and a physical subsystem  104 . Further, medical imaging system  100  also includes one or more system managers  106 , a subsystem controller  108  and  110  associated with each physical subsystem  102  and  104 , and a communication link  112 . The physical subsystems  102  and  104  include, for example, physical processing or operating components or entities that perform different operations in medical imaging system  100 , and in various embodiments are implemented as, but not limited, physical computer systems running on different operating systems. For example, in an X-ray machine, the physical subsystems included are an X-ray generator, a table positioner, a system control and an operator console. 
     In various embodiments, a plurality of subsystem controllers, for example, subsystem controller  108  and subsystem controller  110  may be provided in physical subsystem  102  and physical subsystem  104 , respectively. System manager  106 , subsystem controller  108 , and subsystem controller  110  may be software components or algorithms that interact with each other for use in recovering medical imaging system  100  and/or physical subsystem  102 , and physical subsystem  104  from faults or errors. Further, the functioning, components, and the properties of subsystem controller  108  and subsystem controller  110  may be the same or different as desired or needed. Subsystem controller  108  and subsystem controller  110  ensure that all software components in physical subsystem  102  and physical subsystem  104  respectively, continue to function correctly. System manager  106  ensures that all the subsystem controllers that are a part of its system continue to function correctly. This hierarchy ensures that all subsystems controllers monitor their subsystems while system manager  106  ensures that all the subsystem controllers continue to operate correctly. 
     In various embodiments of the invention, the communication link in medical imaging system  100  between physical subsystem  102 , physical subsystem  104 , and system manager  106  is provided via communication link  112 . Communication link  112  enables platform (e.g., operating system) independent interoperability between a plurality of physical subsystems. In various embodiments, communication link  112  is, for example, enabled through a Shared Memory Communication, or a Common Object Request Broker Architecture (CORBA®) based architecture. In the CORBA based architecture, for example, subsystem controller  108  and system manager  106  expose interfaces through which the Interface Definition Language (IDL) using direct method calls can be made for communication. 
       FIG. 2  shows various components of physical subsystem  102  and physical subsystem  104 , and the logical links between these components. In various embodiments of the invention, physical subsystem  102  includes an actor  202 , and an actor  204 , and physical subsystem  104  includes an actor  206 , an actor  208 , and an actor  210 . In various embodiments, an actor may be, for example, a software component that may be algorithmic or may be a wrapper object for a physical hardware device. In addition, an actor also may store information such that it is able to return to an existing stable state as described in more detail herein. Such information, may include, for example, the current stable state of an actor and persistent data associated with this state. Persistent data may include, for example, data relevant for the actor to rollback or return to the state at which it was prior to the fault. Such data may include, for example, the application protocols selected, etc. Further, such information may be stored, for example, in a secondary storage medium such as hard disk, Storage Area Network (SAN), Redundant Array of Independent Disks (RAID) data storage. 
     It should be noted that, for example, in an Atlas system, a unique set of entities may be included to provide the fundamental functions of the system and provide means to perform these functions such as is described in U.S. Patent Application Publication No. US 2003/0135389 and entitled “Method and Apparatus for Managing A Distributed X-Ray System.” These functional entities may be referred to as Atlas Actors or actors. Actors cooperate to achieve general system operation required by an application. For example, an image detector actor converts an X-ray pattern resulting from the interaction of the X-ray beam with the patient, into a usable information object referred to as an image. As another example, an Acquisition Synchronizer actor coordinates the actions of the different actors in the system. 
     It should be noted that, for example, actors in a system, such as in an Atlas system as described in co-pending U.S. patent application Ser. No. 2003/0135389 A1, entitled “Method and Apparatus for Managing a Distributed X-ray System,” the entire disclosure of which is hereby incorporated by reference herein, may be provided by a unique set of entities that provide the fundamental functions of the system and provide means to perform these functions. These functional entities may be referred to as Atlas Actors or actors. Actors cooperate to achieve general system operation required by an application. For example, an image detector actor converts an X-ray pattern resulting from the interaction of the X-ray beam with the patient, into a usable information object referred to as an image. As another example, an Acquisition Synchronizer actor coordinates the actions of the different actors in the system. 
     Further, actor  204  and actor  206 , and actor  208  and  210  operate to form a logical subsystem  216  and a logical subsystem  218 , respectively. A logical subsystem may be configured as, for example, a functional module containing one or more actors that are interdependent in order to perform a specific action in medical imaging system  100 . The functionality, for example, may be an operator console functionality of X-ray scanner. Examples of functional modules as are known include, Athena the Image Processing Subsystem, Saber the Positioner Subsystem, Calypso the System Control Subsystem. Additionally, it should be noted that the logical subsystems may be provided across physical subsystems. 
     In various embodiments of the invention, the functioning of actor  202  and actor  204  is monitored by subsystem controller  108 , and the functioning of actor  206 , actor  208 , and actor  210  is monitored by subsystem controller  110 . In addition, if a fault occurs in the respective actors, subsystem controller  108  and subsystem controller  110  also notify, for example, actors dependent on the actor having the fault and log an appropriate error. 
     Further, in various embodiments, physical subsystem  102  and physical subsystem  104  further include a user interface  212  and a user interface  214 , respectively. User interface  212  and user interface  214  are configured to receive user inputs and communicate the inputs to actor  204  and actor  210 , respectively, for processing. The actors also may be linked to devices, for example, actor  202  is linked to device  220 . Examples of devices  220  include, but are not limited to, an exposure regulating device, an input device controlling the height of an exam table, a printer, a positioner, a detector, etc. 
       FIG. 3  shows the subcomponents of subsystem controller  108 . It should be noted that the same subcomponents may also be provided in subsystem controller  110 . In various embodiments of the invention, subsystem controller  108  includes a subsystem manager  302  and a location registration component  304 . Subsystem manager  302  monitors the functioning of the actors, for example, actor  202  and actor  204 . The functioning is monitored by a system control mechanism, wherein subsystem manager  302  pings (e.g., sends a request for a response) the respective actors and waits for their response. If a response is not received, subsystem manager  302  may communicate with system controller  106  and generate an appropriate fault recovery mechanism as described in more detail herein with reference to  FIG. 4 . In various embodiments of the invention, subsystem manager  302  also logs any errors and provides appropriate commands or control actions (e.g., to stop operation corresponding to an actor). The commands or control actions may be generated, for example, based on the severity of the error. For example, if a less critical fault occurs in actor  202 , subsystem manager  302  may only restart actor  202 . However, if a more severe fault occurs, subsystem manager  302  may communicate a reboot command to system manager  106 , and system manager  106 , in response, reboots medical imaging system  100 . Subsystem manager  302  also may restore or reset the component that has a fault to a stable state (e.g., system idle application setup, and/or exam-started stable state). 
     In one exemplary embodiment of the invention, one location registration component may be provided for each physical subsystem (e.g., one location registration component  304  present in physical subsystem  102 ). Location registration component  304  registers actor  202  and actor  204  with their functionalities and their dependencies and interdependencies on other actors in medical imaging system  100 . Location registration component  304  also communicates with other location registration components in physical subsystem  104  to generate logical connections. For example, location registration component  304  may communicate with corresponding location registration component  304  (not shown) in physical subsystem  104  to form a logical connection between actor  204  and actor  206 . In an alternate embodiment of the invention, there may be a single location registration component  304  provided for the entire medical imaging system  100 , which may register all the actors with their dependencies and interdependencies. 
     In various embodiments of the invention, fault recovery is based on different levels of fault identified by the subsystem controllers, including, for example, subsystem controller  108  and subsystem controller  110 . The faults relating to each actor in different states are identified and stored in a pre-defined configuration file, which is used by a subsystem controller to determine how to recover from the fault.  FIG. 4  shows an exemplary illustration of the possible states for actor  202  and actor  204 . In an idle state, for example, a fault in actor  202  does not affect the operation of medical imaging system  100 . Thus, the medical imaging system  100  can operate in a degraded mode while actor  202  is reset (e.g., actor  202  restarting itself). As a result, the corresponding severity level value for actor  202  in idle state is stored in the pre-defined configuration file. For example, for an error that allows a degraded mode of operation, the severity level may be, for example, a “4” on a scale of 1 to 4, with “1” being the least severe and “4” being the most severe. In another case, in any state for actor  204 , if the severity of the fault is high, a system level restart is required for all the states defined. As a result, the corresponding severity level value for actor  204  in all defined states is stored in a pre-defined configuration as “1.” 
     In various embodiments of the invention, four possible severity levels may be provided and defined (e.g., defining the corresponding action and operation of the system corresponding to the severity level). It should be noted that the various embodiments are not limited to four severity levels, but may include more or less, as desired or needed (e.g., based on the medical imaging system). Additionally, the responses or actions to be taken for each severity level may be modified as desired or needed (e.g., based upon the type of application corresponding to an actor). In one exemplary embodiment, severity level  1  may be defined as, but not limited to, a fault or error requiring restarting of medical imaging system  100 . This state may occur if, for example, it is determined that as a result in a fault in an actor, the hardware as well as the operating system of medical imaging system  100  require a reboot. The different levels and corresponding actions to be taken may be determined, for example, based on operating conditions, the type of system, system level testing, historical data, etc. Severity level  2  may be defined as, but not limited to, a fault or error requiring an application level reset. The actor having a fault corresponding to a severity level of “2” is restarted and all its related, dependent and/or interdependent actors are reinitialized or restarted. Severity level  3  may be defined as, but not limited to, a fault or error requiring only an actor level reset. The actor having the severity level “3” fault is restarted, while the related, dependent and/or interdependent actors are notified of the fault and their operations may be halted. The severity level  4  may be defined as, but not limited to, a fault or error allowing a degraded mode operation. The actor having the severity level “4” fault is restarted, while the related, dependent and/or interdependent actors, as well as the other components of medical imaging system  100  operate in a degraded mode. The degraded mode may include, for example, a mode of operation wherein the CPU time is provided partially to physical subsystem  102  and physical subsystem  104  for performing operations. The remaining CPU time is utilized for returning medical imaging system  100  to a stable state of operation. 
       FIG. 5  shows an exemplary configuration file that assigns severity values to, for example, actor  202  and actor  204  in accordance with an exemplary embodiment of the invention and that may be implemented by medical imaging system  100 . The file defines a fault recovery function “FAULT_RECOVERY.” Specifically, the function receives input values such as, the ping time-interval, the name of the actors (e.g., actor  202  and actor  204 ), the stable states of medical imaging system  100  (e.g., idle, application setup, exam-started). Further, based on the response of actor  202  and actor  204  in various states, fault severity values may be defined. For example, a fault value “{IDLE, Actor 202 , SEV_ 3 }” relates to severity level  3  response of fault in actor  202  in the idle state. The configuration file includes mapping for each type of fault thatmay be present in medical imaging system  100 . As described above, the configuration file captures the state in which a fault may potentially occur for an actor and the type of recovery to be performed based on the fault. 
       FIG. 6  is a flowchart of a process  600  for fault handling and recovery of a medical imaging system  100  in accordance with an exemplary embodiment of the invention. The flowchart is described in connection with physical subsystem  102  and its subcomponents. However, it should be noted the process of the flowchart may be implemented in connection with other components of medical imaging system, such as, for example, physical subsystem  104  and other subsystems of various embodiments of medical imaging system  100 . Specifically, at  602 , subsystem controller  108  pings the registered actors (e.g., as registered in location registration component  304 ), for example, actor  202  and actor  204 . In various embodiments, the ping request is sent after a predefined time interval, which may vary based on the requirements of medical imaging system  100 . At  604 , subsystem controller  108  then waits for a response from actor  202  and actor  204 . If a response is received, at  606 , subsystem controller  108  waits for the next ping request. This process is repeated until a response is not received form actor  202  and actor  204 . For example, subsystem controller  108  iteratively goes back to  602  and pings these actors. 
     At  604 , if no response is received from actor  202  and/or actor  204 , the at  608 , all zombie or hung processes for the non-responding actor are terminated by subsystem controller  108 . This step is performed to halt any operations or functions being performed by the non-responding actor. 
     After terminating the actors not responding, subsystem controller  108 , at step  610 , retrieves information regarding the state of medical imaging system  100 , and the corresponding severity level of the fault in that state, for example, from a table stored in a memory of medical imaging system  100 . The information regarding the severity level of the fault in a particular state may be retrieved, for example, from the configuration file. Based on the severity level of the fault, at  612 , subsystem controller  108  determines if the actor having the fault has severity level  4  in that particular state of medical imaging system  100 . If the severity level is 4, at  614 , subsystem controllers  108  notifies system manager  106  that a degraded mode of operation is required. At  616 , in response to this notification, system manager  106  notifies users on user interface  212  and user interface  214  regarding the fault. At  618 , system manager  106  then sets medical imaging system  100  in a degraded mode of operation. At  619 , only the actor having the fault is restarted or reinitialized, while the other actors in medical imaging system  100  perform their operations. In the degraded mode, for example, the other actors and their operations may be allocated lesser CPU time. The remaining CPU time is used to restore medical imaging system  100  to a stable state. 
     At  612 , if subsystem controller  108  determines that the fault is not of severity level  4 , at  620 , it determines if the fault is a severity level  3  fault. If the fault is of a severity level  3 , then at  622  subsystem controller  108  notifies system manager  106  of the fault, and in response system manager  106  notifies all the related actors of the actor(s) having this fault to halt operation. In addition, at  624 , the actor having the fault is restarted or reinitialized. Thus, at  622 , the functioning of related actors is halted so that no error is developed in their functioning due to an error in the actor having the fault. After the actor having a fault is restarted or reinitialized, the halted actors return to normal operation in a stable state. In addition, system controller  108  returns to  602  and performs the process  600  again iteratively. 
     At  620 , if subsystem controller  108  determines that the fault is not of severity level  3 , at  626 , it determines if the fault is a severity level  2  fault. If the fault is of a severity level  3 , at  628  subsystem controller  108  notifies system manager  106  of the fault, and in response system manager  106  notifies all the related actors of the actors having a fault to restart or reinitialize. In addition, at  628 , the actor having the fault is restarted. While reinitializing, the related actors store their current state and receive input values from the actor having the fault. After the related actor have restarted or reinitialized themselves, medial imaging system  100  returns to a stable state and subsystem controller  108  return to  602  and performs the process  600  again iteratively. 
     At  626 , if subsystem controller  108  determines that the fault is not of severity level 2, at  632 , subsystem controller  108  notifies system manager  106  of a system level restart. At  634 , system manager  106  notifies the users on user interface  212  and user interface  214  regarding rebooting medical imaging system  100 . In a system level reset, the operating system of physical subsystem  102  and physical subsystem  104  are rebooted and medical imaging system  100  starts its operations again. 
     Various embodiments of the invention provide higher system availability. Recovering the system from faults that at times may not require rebooting the system provides the higher system availability. Thus, a higher number of imaging procedures may be performed per day. 
     Further, various embodiments of the invention provide higher software reliability. In various embodiments of the invention, the fault management and recovery architecture tracks the fault from its source and restores the component having the fault. Further, various embodiments of the invention provide a configurable medical imaging system. The response of the medical imaging system to a fault can be varied based on the severity of the fault that may occur. 
     A technical effect of various embodiments of the invention is to categorize the faults based on severity and control the operation of the hardware of the medical imaging system based on the severity of the fault. This reduces system rebooting and provides more system uptime. Another technical effect of various embodiments of the invention to perform recovery of software components over a distributed environment. In a distributed environment, actors of a logical subsystem may be distributed over different physical subsystems. The various embodiments of the invention perform recovery of these related actors over different physical subsystems. 
     The computer system for implementing the various embodiments as part of a medical imaging system may include a computer, an input device, a display unit and an interface, for example, for accessing the Internet. The computer may include a microprocessor. The microprocessor may be connected to a communication bus. The computer may also include a memory. The memory may include Random Access Memory (RAM) and Read Only Memory (ROM). The computer system further may include a storage device, which may be a hard disk drive or a removable storage drive such as a floppy disk drive, optical disk drive, and the like. The storage device also may be other similar means for loading computer programs or other instructions into the computer system. 
     The computer system executes a set of instructions that are stored in one or more storage elements, in order to process input data. The storage elements may also hold data or other information as desired or needed. The storage element may be in the form of an information source or a physical memory element within the processing machine. 
     The set of instructions may include various commands that instruct the processing machine to perform specific operations such as the processes of the various embodiments of the invention. The set of instructions may be in the form of a software program. The software may be in various forms such as system software or application software. Further, the software may be in the form of a collection of separate programs, a program module within a larger program or a portion of a program module. The software also may include modular programming in the form of object-oriented programming. The processing of input data by the processing machine may be in response to user commands, or in response to results of previous processing, or in response to a request made by another processing machine. 
     While the invention has been described in terms of various specific embodiments, those skilled in the art will recognize that the invention can be practiced with modification within the spirit and scope of the claims.