English:
[0001] Automated diagnostic systems utilizing, for example, machine learning, have been playing an increasingly important role in healthcare. Over the last few years, machine learning techniques (especially neural networks or deep neural networks) have been successfully applied to medical image classification. Classification modules may be used to provide an indication for the presence of a certain anatomy, pathology, object and/or organ in an image, but do not provide information with respect to a spatial location of the identified classification. Although some techniques for generating visual explanations associated with an output of a e.g. deep neural network classifier have been proposed, these methods provide means for measuring the impact of individual input voxels on the classifier decision. In some cases, however, these methods are limited in their practical applicability as resulting attribution heat maps may be diffuse and difficult to interpret.
[0002] The exemplary embodiments are directed to a computer-implemented method of training a machine learning module to provide classification and localization information for an image study, comprising: receiving a current image study; applying the machine learning module to the current image study to generate a classification result including a prediction for one or more class labels for the current image study using a classification module of the machine learning module; receiving, via a user interface, a user input indicating a spatial location corresponding to a predicted class label; and training a localization module of the machine learning module using the user input indicating the spatial location corresponding to the predicted class label.
[0003] The exemplary embodiments are directed to a system of training a machine learning module to provide classification and localization information for an image study, comprising: a non-transitory computer readable storage medium storing an executable program; and a processor executing the executable program to cause the processor to: receive a current image study; apply the machine learning module to the current image study to generate a classification result including a prediction for one or more class labels for the current image study using a classification module of the machine learning module; receive, via a user interface, a user input indicating a spatial location corresponding to a predicted class label; and train a localization module of the machine learning module using the user input indicating the spatial location corresponding to the predicted class label.
[0004] The exemplary embodiments are directed to a non-transitory computer-readable storage medium including a set of instructions executable by a processor, the set of instructions, when executed by the processor, causing the processor to perform operations, comprising: receiving a current image study; applying a machine learning module to the current image study to generate a classification result including a prediction for one or more class labels for the current image study using a classification module of the machine learning module; receiving, via a user interface, a user input indicating a spatial location corresponding to a predicted class label; and training a localization module of the machine learning module using the user input indicating the spatial location corresponding to the predicted class label.
[0005] Fig. 1 shows a schematic diagram of a system according to an exemplary embodiment.
[0006] Fig. 2 shows another schematic diagram of the system according to Fig. 1.
[0007] Fig. 3 shows a schematic user interface according to an exemplary embodiment.
[0008] Fig. 4 shows another schematic user interface according to an exemplary embodiment.
[0009] Fig. 5 shows a flow diagram of a method according to an exemplary embodiment.
[0010] The exemplary embodiments may be further understood with reference to the following description and the appended drawings, wherein like elements are referred to with the same reference numerals. The exemplary embodiments relate to systems and methods for machine learning and, in particular, relate to systems and methods for dynamically extending and/or modifying a machine learning module. The machine learning module comprises a pretrained classification module, which identifies a class label for a particular image study, and an untrained or partially trained localization module, which is to be trained using relevant spatial information provided by a user based on the identified class label and/or the image study. Thus, once the localization module has been trained to a stable state, the machine learning module may autonomously provide both a class label and a relevant spatial location for an image study. The classification module may also be configured to continually adapt based on other user input such as, for example, the addition of new classes and/or corrections to an identified class label. It will be understood by those of skill in the art that although the exemplary embodiments are shown and described with respect to X-ray images or image studies, the systems and methods of the present disclosure may be similarly applied to any of a variety of medical imaging modalities in any of a variety of medical fields for any of a variety of different pathologies and/or target areas of the body.
[0011] As shown in Fig. 1, a system 100, according to an exemplary embodiment of the present disclosure, applies a classification module to an image study to provide a classification decision for the image study to a user (e.g., clinician). The user may then input relevant information based on the image study and/or the classification decision. This relevant information, along with subsequent relevant information for subsequent image studies, may be used to train a localization module and/or continually adapt the classification module, as will be described below. The system 100 comprises a processor 102, a user interface 104, a display 106 and a memory 108. The processor 102 may comprise a machine learning module 110 and a training engine 116 for training the machine learning module 110. The machine learning module 110 is, for example, a deep learning network. There are various types of applicable deep learning networks as known in the art. However, other suitable or comparable types machine learning techniques may be used as would be understood by one having ordinary skill in the art. The machine learning module 110 may further include a classification module 112 and a localization module 114. The classification module 112 may be applied to a current image 118, which may be received and stored to the memory 108, to generate a classification and/or localization result 122 for the current image study 118 to the user via, for example, the display 106. Suitable techniques for the classification module 112 include, for example, deep learning techniques such as convolutional neural networks (e.g., densely connected neural networks, residual neural networks, networks resulting from architecture search algorithms, capsule networks etc.). Alternatively, techniques based on image descriptors (e.g. HOG, SURF, SIFT, Eigen-Features ... )and other machine learning techniques can be employed. The user may input, via the user interface 104, relevant information such as, for example, a bounding box showing a relevant spatial location of an identified classification (e.g., pathology, organ, object etc.), a new class for the classification module, and/or corrections to the classification decision for the current image study 118. This relevant information (e.g., user input) is added to a database 120 in the memory 108, which may be used by, for example, the training engine 116 for training one of the localization module 114 and/or classification module 112 of the machine learning module 110, as will be described in further detail below. Suitable techniques for the localization module include e.g. methods from the field of object detection / instance segmentation such as fast region-based convolutional neural networks, ’’you only look once” architectures, RetinaNets or Mask R-CNNs. Similarly, classification-based detectors (e.g. sliding windows methods), or voting-based techniques (e.g. Generalized Hough Transform, Hough Forest, etc.) can be employed.
[0012] In some embodiments, the classification module 112 of the machine learning module 110 has been pre-trained, during manufacturing, with training data including image studies (e.g., x-ray images or image studies) that have corresponding classification information so that the machine learning module 110 is delivered to a clinical site (e.g., hospital) with classification capabilities. Thus, the classification module 112 is trained to provide a medical image classification (e.g., class label) based on an image being analyzed. Image classifications provide, for example, an indication of a presence of a particular anatomy, pathology, object, organ, etc. Classes may include, for example, the presence of effusion, fractures, nodules, support devices, etc. Although the classification module 112 has been pre-trained, the classification module 112 may be configured to continually adapt by learning new user inputs such as, for example, new classes and/or classification corrections. In some embodiments, the classification module 112 may include an internal module such as, for example, an image classification module.
[0013] While the classification module 112 is pre-trained, the localization module 114 may be manufactured and delivered to the clinical site in an untrained state. Thus, with each use of the machine learning module 110, user input including spatial location information may be used to train the localization module 114 so that once the localization module is trained to a stable state, the localization module 114 will be capable of identifying a relevant spatial location of an identified class for a particular image study. In some embodiments, user inputs indicating relevant spatial information may include, for example, a bounding box drawn over a relevant portion of the image study. In some embodiments, the localization module 114 may include an internal module for bounding box detection. It will be understood by those of skill in the art that although the localization module 114 is described as being manufactured and delivered in an untrained state, the localization module 114 may also be delivered in a partially trained state using, for example, testing data acquired during a testing stage. With the acquisition of sufficient data and subsequent training, the machine learning module 110 may eventually be a fully trained, autonomous decision making system.
[0014] The user may input any relevant information via, the user interface 104, which may include any of a variety of input devices such as, for example, a mouse, a keyboard and/or a touch screen via the display 106. User inputs may be stored to the database 120 for training of the classification module 112 and/or localization module 114.
[0015] As shown in Fig. 2, the current image study 118, which requires an assessment/diagnosis, is directed to the machine learning module 110 so that the classification and localization results 122 based on the application of the classification module 112 and the localization module 114 are displayed. As described above, during earlier iterations of the machine learning module 110 in which the localization module 114 has not been trained to a stable state, the current image study 118 may be displayed to the user along with the classification result. Based on the displayed current image study 118 and/or the classification result for the current image study 118, the user may indicate a relevant spatial location by, for example, drawing a bounding box over a relevant portion of the displayed current image study 118.
[0016] The system 100 may keep track of labels for which the classification module 112 or localization module 114 is in a stable state. To determine whether a module is considered as stable for a certain label, the system 100 may rely on a set of predefined performance requirements and/or rules. An exemplary rule may be that at least 500 images containing the label were seen during on-site module adaptation. However, it should be understood that this is just one example of a predefined requirement/rule and other requirements and/or rules may also be used. Classification or localization results related to stable classes are forwarded to the user interface. Classification or localization results related to labels which are not considered to be stable may not be directly displayed to the user.
[0017] Fig. 3 shows an exemplary embodiment of a user interface displaying a classification and localization result for a current image study. In this example, the localization module 114 has not yet been trained to a stable state (e.g., trained to meet predetermined performance requirements) for at least one of the identified class labels. Where the localization module has not been trained to a steady state for a class label, the current image study is displayed to the user alongside the classification result so that the user may input relevant spatial location information such as, for example, a bounding box. The bounding box may be sized and positioned, as desired. Identified class labels may be selected by the user, as desired, to view any identified spatial locations (if stable) and/or input relevant spatial location for that class label (if unstable). Along with the automated classification results displayed to the user, the user interface may include options such as, for example adding a bounding box (or other relevant visual spatial location indication) to show a spatial location of a particular class indication, adding additional findings (e.g., additional class labels), and removing findings. It will be understood by those of skill in the art that the user interface may include other menu options related to the classification/localization result.
[0019] It will be understood by those of skill in the art that the user interfaces described and shown in Figs. 3 and 4 are exemplary only. User interfaces may have any of a variety of configurations and include any of a variety of user options which may be displayed in any of a variety of ways so long as the classification/localization results are displayed to the user thereby. The user may edit either the localization result and/or the classification result, as desired.
[0020] Any user inputs such as, for example, relevant spatial location, edits, additions or corrections may be stored to the database 120 to be used by the training engine 116 to train the classification module 112 and/or localization module 114 accordingly.
[0021] Those skilled in the art will understand that the classification module 112 and the localization module 114 of the machine learning module 110 along with the training engine 116 may be implemented by the processor 102 as, for example, lines of code that are executed by the processor 102, as firmware executed by the processor 102, as a function of the processor 102 being an application specific integrated circuit (ASIC), etc. It will also be understood by those of skill in the art that although the system 100 is shown and described as comprising a computing system comprising a single processor 102, user interface 104, display 106 and memory 108, the system 100 may be comprised of a network of computing systems, each of which includes one or more of the components described above. In one example, the classification module 112 and the localization module 114 of the machine learning module 110 along with the training engine 116 may be executed via a central processor of a network, which is accessible via a number of different user stations. Alternatively, one or more of the classification module 112 and the localization module 114 of the machine learning module 110 along with the training engine 116 may be executed via one or more processors. Similarly, the database 120 may be stored to a central memory 108. The current image study 118 may be acquired from any of a plurality of imaging devices networked with or otherwise connected to the system 100 and stored to a central memory 108 or, alternatively, to one or more remote and/or network memories 108.
[0022] Fig. 5 shows an exemplary method 200 for providing classification/localization results for a current image study 118 and using user inputs to train a localization module 114 and/or a classification module 112 to expand and/or adapt a machine learning module 110 according to the system 100. In 210, the current image study 118 is received and/or stored to the memory 108 so that the machine learning module 110 may be applied to the current image study 118 in 220. Using the classification module 112 and the localization module 114, the machine learning module 110 provides a classification/localization result 122 to the user in 230. Classification results may include predictions of one or more findings including one or more class labels, which indicate a presence (or absence) of, for example, certain anatomies, pathologies, objects, or organs. Localization results may include a visual display of a spatial location of the predicted (e.g., identified as present) class labels. In 240, the user may provide user input, via the user interface 104, based on the classification/localization result 122.
[0023] As described above, however, for earlier iterations of the machine learning module 110, while the classification module 112 is pre-trained to be able to provide classifications (e.g., identify class labels) for the current image study 118, the localization module 114 may be untrained or partially trained so that the machine learning module 110 is not yet trained to show relevant spatial location information. Thus, in these cases, a user interface may show the current image study 118 along with the classification results so that the user input may include relevant spatial information via, for example, a bounding box drawn over a relevant portion of the current image study 118.
[0024] In later iterations, where the localization module 114 has been trained to a stable state, the classification/localization result 122 will identify relevant class labels and show relevant spatial locations for corresponding identified classes. In these embodiments, the user input may include editing of spatial information by, for example, adjusting a location and/or size of a displayed bounding box. Regardless of whether the localization module 114 is in a stable state, however, user inputs may also include other data such as, for example, adding findings (e.g., addition of class labels) and/or corrections to findings (e.g., removing findings or class labels).
[0025] In 250, all the user inputs are stored to the database 120 so that, in 260, the training engine 116 trains the machine learning module 110 to include the data from the database 120. In particular, the classification module 112 is trained with user inputs corresponding to classification results while the localization module is trained with user inputs corresponding to spatial location. The classification module 112 and the localization module 114, however, implement transfer-learning techniques (e.g., sharing of module components, sharing of feature maps) in order to exploit the commonalities of localization and classification tasks. For example, certain feature extractors or convolutional filters may be shared among both the classification module 112 and the localization module 114.
[0026] In some embodiments, the classification module 112 and the localization module 114 are deep neural networks and share the same layers as a backbone for an object detector. In other embodiments, only certain layers of the classification network and object detector backbone may be shared. In further embodiments, it is possible to implement the training setup in such a way that the classification and localization modules 112, 114 are updated in an alternating fashion. If the classification and localization modules 112, 114 share components, the training process may be configured in such a way that during the retraining of individual modules, certain layers/components (e.g., neural network convolutional filter weights) may be frozen. For example, during a gradient step with respect to the classification loss, a latter half of the layers of a shared deep neural network may be frozen while during a gradient step with respect to an object localization loss, a first half of the layers may be frozen. In other embodiments, it is also possible to implement the training setup in such a way that the classification and localization modules 112, 114 are jointly updated (e.g., by combining the classification and localization loss functionals and performing a joint backpropagation).
[0027] It will be understood by those of skill in the art that the method 200 may be continuously repeated so that machine learning module 110 is dynamically expanded and modified with each use thereof. In particular, since the localization module 114 is continuously trained with new localization data provided by the user, the localization module 114 will eventually be trained to a stable state so that the deep neural network 110 may provide a fully autonomous classification and localization result for an image study. Even when the deep neural network 110 is capable of providing a fully autonomous result, however, user input may be utilized to continually adapt and modify the deep neural network 110 to overcome shifts in data distribution (“domain bias”) and to mitigate the effect of catastrophic forgetting. An on-site adaption may be continued to be triggered based on a set of pre-defined rules (e.g., 1000 new images containing at least 10000 foreground/positive labels are available).
[0028] Those skilled in the art will understand that the above-described exemplary embodiments may be implemented in any number of manners, including, as a separate software module, as a combination of hardware and software, etc. For example, the machine learning module 110, classification module 112, localization module 114 and training engine 116 may be programs including lines of code that, when compiled, may be executed on the processor 102.
[0029] Although this application described various embodiments each having different features in various combinations, those skilled in the art will understand that any of the features of one embodiment may be combined with the features of the other embodiments in any manner not specifically disclaimed or which is not functionally or logically inconsistent with the operation of the device or the stated functions of the disclosed embodiments.
[0030] It will be apparent to those skilled in the art that various modifications may be made to the disclosed exemplary embodiments and methods and alternatives without departing from the spirit or scope of the disclosure. Thus, it is intended that the present disclosure cover the modifications and variations provided that they come within the scope of the appended claims and their equivalents.

中文:
[0001]利用例如机器学习的自动诊断系统在健康护理中发挥着越来越重要的作用。在过去的几年里,机器学习技术(尤其是神经网络或深度神经网络)已经成功地应用于医学图像分类。分类模块可以用于提供对图像中存在特定解剖结构、病理、对象和/或器官的指示,但是不提供关于所识别的分类的空间位置的信息。尽管已经提出了一些用于生成与例如深度神经网络分类器的输出相关联的视觉解释的技术,但是这些方法提供了用于测量个体输入体素对分类器决策的影响的手段。然而,在某些情况下,这些方法的实际适用性有限,因为所得到的归因热图可能是分散的,难以解读。
[0002]示例性实施例涉及一种训练机器学习模块以提供针对图像研究的分类和定位信息的计算机实施的方法,包括:接收当前图像研究;将所述机器学习模块应用于所述当前图像研究,以使用所述机器学习模块的分类模块来生成分类结果,所述分类结果包括对所述当前图像研究的一个或多个类别标签的预测;经由用户接口接收指示与预测的类别标签相对应的空间位置的用户输入;并且使用指示与预测的类别标签相对应的空间位置的所述用户输入来训练所述机器学习模块的定位模块。
[0003]示例性实施例涉及一种训练机器学习模块以提供针对图像研究的分类和定位信息的系统,包括:非瞬态计算机可读存储介质,其存储可执行程序;以及处理器,其执行所述可执行程序以使所述处理器:接收当前图像研究;将所述机器学习模块应用于所述当前图像研究,以使用所述机器学习模块的分类模块来生成分类结果,所述分类结果包括对所述当前图像研究的一个或多个类别标签的预测;经由用户接口接收指示与预测的类别标签相对应的空间位置的用户输入;并且使用指示与预测的类别标签相对应的空间位置的所述用户输入来训练所述机器学习模块的定位模块。
[0004]示例性实施例涉及一种非瞬态计算机可读存储介质,包括可由处理器执行的指令集,所述指令集在由所述处理器执行时使所述处理器执行操作,所述操作包括:接收当前图像研究;将机器学习模块应用于所述当前图像研究,以使用所述机器学习模块的分类模块来生成分类结果,所述分类结果包括对所述当前图像研究的一个或多个类别标签的预测;经由用户接口接收指示与预测的类别标签相对应的空间位置的用户输入;并且使用指示与预测的类别标签相对应的空间位置的所述用户输入来训练所述机器学习模块的定位模块。
[0005]图1示出了根据示例性实施例的系统的示意图。
[0006]图2示出了根据图1的系统的另一示意图。
[0007]图3示出了根据示例性实施例的示意性用户接口。
[0008]图4示出了根据示例性实施例的另一示意性用户接口。
[0009]图5示出了根据示例性实施例的方法的流程图。
[0010]参考以下描述和附图可以进一步理解示例性实施例,其中,相同的元件用相同的附图标记表示。示例性实施例涉及用于机器学习的系统和方法,具体地涉及用于动态地扩展和/或修改机器学习模块的系统和方法。所述机器学习模块包括预先训练的分类模块和未训练或部分训练的定位模块,所述预先训练的分类模块识别针对特定图像研究的类别标签;所述未训练或部分训练的定位模块将使用用户基于所识别的类别标签和/或图像研究提供的相关空间信息来训练。因此,一旦定位模块已经被训练到稳定状态,机器学习模块就可以自主地提供针对图像研究的类别标签和相关空间位置两者。分类模块还可以被配置为基于其他用户输入(例如,新类别的添加和/或对所识别的类别标签的校正)来连续地进行适配。本领域技术人员将理解,尽管示例性实施例是关于X射线图像或图像研究示出和描述的,但是本公开的系统和方法可以类似地应用于各种医学领域中的任何一种中的各种医学成像模态中的任何一种以用于身体的各种不同病理和/或目标区域中的任何一种。
[0011]如图1所示,根据本公开的示例性实施例的系统100将分类模块应用于图像研究,以向用户(例如,临床医生)提供针对图像研究的分类决策。然后,用户可以基于图像研究和/或分类决策来输入相关信息。该相关信息以及用于后续图像研究的后续相关信息可以用于训练定位模块和/或连续地适配分类模块,如下面将描述的。系统100包括处理器102、用户接口104、显示器106和存储器108。处理器102可以包括机器学习模块110和用于训练机器学习模块110的训练引擎116。机器学习模块110例如是深度学习网络。存在本领域中已知的各种类型的可应用的深度学习网络。然而,如本领域技术人员将理解的,可以使用其他合适或类似类型的机器学习技术。机器学习模块110还可以包括分类模块112和定位模块114。分类模块112可以被应用于当前图像118(其可以被接收并存储到存储器108),以经由例如显示器106向用户生成针对当前图像研究118的分类和/或定位结果122。用于分类模块112的合适技术包括例如深度学习技术,例如卷积神经网络(例如,密集连接的神经网络、残差神经网络、由架构搜索算法产生的网络、胶囊网络)。替代地,可以采用基于图像描述符的技术(例如,HOG、SURF、SIFT、Eigen-Features...)和其他机器学习技术。用户可以经由用户接口104输入相关信息,例如示出所识别的分类(例如,病理、器官、对象等)的相关空间位置的边界框、用于分类模块的新类别和/或对针对当前图像研究118的分类决策的校正。该相关信息(例如,用户输入)被添加到存储器108中的数据库120,例如,训练引擎116可以使用该相关信息来训练机器学习模块110的定位模块114和/或分类模块112中的一个,如下面将进一步详细描述的。用于定位模块的合适技术包括例如来自对象检测/实例分割领域的方法,例如基于快速区域的卷积神经网络、“You Only Look Once”架构、RetinaNets或Mask R-CNNs。类似地,可以采用基于分类的检测器(例如滑动窗口方法)或基于投票的技术(例如,Generalized Hough Transform、Hough Forest等)。
[0012]在一些实施例中,机器学习模块110的分类模块112在制造期间已经利用包括具有对应分类信息的图像研究(例如,X射线图像或图像研究)的训练数据进行预训练,使得机器学习模块110被交付到具有分类能力的临床场所(例如,医院)。因此,分类模块112被训练以基于正被分析的图像来提供医学图像分类(例如,类别标签)。图像分类提供例如对特定解剖结构、病理、对象、器官等的存在的指示。类别可以包括例如积液、骨折、结节、支撑设备等的存在。尽管分类模块112已经被预训练,但是分类模块112可以被配置为通过学习新的用户输入(例如,新的类别和/或分类校正)来连续地适配。在一些实施例中,分类模块112可以包括内部模块,例如图像分类模块。
[0013]当分类模块112被预训练时,定位模块114可以以未训练的状态被制造并交付到临床场所。因此,随着机器学习模块110的每次使用,包括空间位置信息的用户输入可以用于训练定位模块114,使得一旦定位模块被训练到稳定状态,定位模块114就将能够识别针对特定图像研究的所识别类别的相关空间位置。在一些实施例中,指示相关空间信息的用户输入可以包括例如在图像研究的相关部分上绘制的边界框。在一些实施例中,定位模块114可以包括用于边界框检测的内部模块。本领域技术人员将理解,尽管定位模块114被描述为以未训练的状态制造和交付,但是定位模块114也可以使用例如在测试阶段期间采集的测试数据以部分训练的状态交付。随着采集足够的数据和后续训练,机器学习模块110最终可以是完全训练的自主决策系统。
[0014]用户可以经由用户接口104输入任何相关信息,用户接口104可以包括各种输入设备中的任何一种,例如鼠标、键盘和/或经由显示器106的触摸屏。用户输入可以被存储到数据库120以用于分类模块112和/或定位模块114的训练。
[0015]如图2所示,需要评估/诊断的当前图像研究118被引导到机器学习模块110,使得基于分类模块112和定位模块114的应用的分类和定位结果122被显示。如上所述,在定位模块114尚未被训练到稳定状态的机器学习模块110的较早迭代期间,当前图像研究118可以与分类结果一起被显示给用户。基于所显示的当前图像研究118和/或针对当前图像研究118的分类结果,用户可以通过例如在所显示的当前图像研究118的相关部分上绘制边界框来指示相关空间位置。
[0016]系统100可以跟踪分类模块112或定位模块114处于稳定状态的标签。为了确定模块对于某个标签是否被认为是稳定的,系统100可以依赖于一组预定义的性能要求和/或规则。示例性规则可以是在现场模块适应期间看到至少500个包含标签的图像。然而,应当理解,这仅仅是预定义的要求/规则的一个示例,并且也可以使用其他要求和/或规则。与稳定类别相关的分类或定位结果被转发到用户接口。与被认为不稳定的标签相关的分类或定位结果可以不直接显示给用户。
[0017]图3示出了显示针对当前图像研究的分类和定位结果的用户接口的示例性实施例。在该示例中,定位模块114尚未针对所识别的类别标签中的至少一个类别标签被训练到稳定状态(例如,被训练为满足预定性能要求)。在定位模块尚未针对类别标签被训练到稳定状态的情况下,当前图像研究与分类结果一起被显示给用户,使得用户可以输入相关的空间位置信息,例如边界框。边界框的尺寸和位置可以根据需要而定。用户可以根据需要选择所识别的类别标签,以查看任何所识别的空间位置(如果稳定的话)和/或输入针对该类别标签的相关空间位置(如果不稳定的话)。除了向用户显示自动分类结果之外,用户接口可以包括例如添加边界框(或其他相关的视觉空间位置指示)以示出特定类别指示的空间位置、添加附加发现(例如,附加类别标签)和移除发现的选项。本领域技术人员将理解,用户接口可以包括与分类/定位结果相关的其他菜单选项。
[0019]本领域技术人员将理解,图3和4中描述和示出的用户接口仅是示例性的。用户接口可以具有各种配置中的任何一种,并且包括可以以各种方式中的任何一种显示的各种用户选项中的任何一种,只要分类/定位结果由此显示给用户即可。用户可以根据需要编辑定位结果和/或分类结果。
[0020]任何用户输入(例如相关的空间位置、编辑、添加或校正)都可以存储到数据库120,以被训练引擎116用于相应地训练分类模块112和/或定位模块114。
[0021]本领域技术人员将理解,机器学习模块110的分类模块112和定位模块114以及训练引擎116可以由处理器102实施为例如由处理器102执行的代码行、由处理器102执行的固件、是专用集成电路(ASIC)的处理器102的功能等等。本领域技术人员还将理解,尽管系统100被示出和描述为包括计算系统,该计算系统包括单个处理器102、用户接口104、显示器106和存储器108,但是系统100可以包括计算系统的网络,每个计算系统包括上述部件中的一个或多个。在一个示例中,机器学习模块110的分类模块112和定位模块114以及训练引擎116可以经由网络的中央处理器来执行,该中央处理器可经由多个不同的用户站访问。替代地,机器学习模块110的分类模块112和定位模块114中的一个或多个以及训练引擎116可以经由一个或多个处理器来执行。类似地,数据库120可以被存储到中央存储器108。当前图像研究118可以从与系统100联网或以其他方式连接到系统100的多个成像设备中的任何一个采集,并且被存储到中央存储器108,或替代地,被存储到一个或多个远程和/或网络存储器108。
[0022]图5示出了根据系统100的用于提供针对当前图像研究118的分类/定位结果并使用用户输入来训练定位模块114和/或分类模块112以扩展和/或适配机器学习模块110的示例性方法200。在210中,当前图像研究118被接收和/或被存储到存储器108,使得在220中机器学习模块110可以被应用于当前图像研究118。在230中,使用分类模块112和定位模块114,机器学习模块110向用户提供分类/定位结果122。分类结果可以包括对包括一个或多个类别标签的一个或多个发现的预测,其指示例如某些解剖结构、病理、对象或器官的存在(或不存在)。定位结果可以包括预测的(例如,被识别为存在的)类别标签的空间位置的视觉显示。在240中,用户可以基于分类/定位结果122经由用户接口104提供用户输入。
[0023]然而,如上所述,对于机器学习模块110的早期迭代,虽然分类模块112被预训练为能够向当前图像研究118提供分类(例如,识别类别标签),但是定位模块114可能是未训练的或部分训练的,使得机器学习模块110尚未被训练为示出相关的空间位置信息。因此,在这些情况下,用户接口可以示出当前图像研究118以及分类结果,使得用户输入可以包括经由例如在当前图像研究118的相关部分上绘制的边界框的相关空间信息。
[0024]在随后的迭代中,在定位模块114已经被训练到稳定状态的情况下,分类/定位结果122将识别相关的类别标签,并且示出所识别的对应类别的相关空间位置。在这些实施例中,用户输入可以包括通过例如调整所显示的边界框的位置和/或尺寸来编辑空间信息。然而,无论定位模块114是否处于稳定状态,用户输入都还可以包括其他数据,例如添加发现(例如,添加类别标签)和/或对发现的校正(例如,移除发现或类别标签)。
[0025]在250中,所有用户输入都被存储到数据库120中,使得在260中,训练引擎116将机器学习模块110训练为包括来自数据库120的数据。特别地,利用与分类结果相对应的用户输入来训练分类模块112,同时利用与空间位置相对应的用户输入来训练定位模块。然而,分类模块112和定位模块114实施迁移学习技术(例如,模块部件的共享、特征图的共享),以便利用定位和分类任务的共性。例如,某些特征提取器或卷积滤波器可以在分类模块112和定位模块114两者之间共享。
[0026]在一些实施例中,分类模块112和定位模块114是深度神经网络,并且共享与对象检测器的主干相同的层。在其他实施例中,可以仅共享分类网络和对象检测器主干的某些层。在另外的实施例中,可以以分类模块112和定位模块114以交替方式更新的这种方式实施训练设置。如果分类模块112和定位模块114共享部件,则训练过程可以以在个体模块的重新训练期间可以冻结某些层/部件(例如,神经网络卷积滤波器权重)的这种方式进行配置。例如,在关于分类损失的梯度步骤期间,可以冻结共享深度神经网络的层的后半部,而在关于对象定位损失的梯度步骤期间,可以冻结所述层的前半部。在其他实施例中,还可以以联合更新分类模块112和定位模块114的方式(例如,通过组合分类和定位损失函数并执行联合反向传播)来实施训练设置。
[0027]本领域技术人员将理解,方法200可以连续地重复,使得机器学习模块110在每次使用时被动态地扩展和修改。特别地,由于利用由用户提供的新定位数据连续地训练定位模块114,因此定位模块114最终将被训练到稳定状态,使得深度神经网络110可以提供针对图像研究的完全自主的分类和定位结果。然而,即使当深度神经网络110能够提供完全自主的结果时,用户输入也可以用于连续地适配和修改深度神经网络110以克服数据分布的偏移(“域偏差”)并减轻灾难性遗忘的影响。可以基于一组预定义规则(例如,包含至少10000个前景/正类标签的1000个新图像可用)继续触发现场适配。
[0028]本领域技术人员将理解,上述示例性实施例可以以多种方式实施,包括被实施为单独的软件模块、硬件和软件的组合等。例如,机器学习模块110、分类模块112、定位模块114和训练引擎116可以是包括代码行的程序,所述代码行在被编译时可以在处理器102上执行。
[0029]尽管本申请以各种组合描述了均具有不同特征的各种实施例,但是本领域技术人员将理解,一个实施例的任何特征可以以未明确放弃或不在功能上或逻辑上与设备的操作或所公开实施例的规定功能不一致的任何方式与其他实施例的特征组合。
[0030]对本领域技术人员显而易见的是,可以对公开的示例性实施例和方法以及替代方案进行各种修改而不脱离本公开的精神或范围。因此,本公开旨在覆盖修改及变型,只要它们落入权利要求书以及与其等价的范围内。