System, method and computer-accessible medium for quantification of blur in digital images

The present disclosure discusses systems and methods to detect blur in digital images. The solution can be incorporated into the quality control systems of pathology and other slide scanners or can be a stand-alone solution. The solution can identify scanned images that include blur and cause the scanner to automatically rescan the blurry image. The solution can also identify regions of the scanned image that include blur. The solution can generate blur maps for each of the scanned images that identify regions of the scanned image that include blur.

BACKGROUND OF THE DISCLOSURE

Pathology review is traditionally conducted with analog (or physical) slides. At a major hospital, the digitization of the slides would require the scanning of tens of thousands of digital slides per month. One bottleneck of high-throughput scanning is quality control (QC). Currently, digital slides are screened manually to detect out of focus regions and other defects in the scanned slides.

SUMMARY OF THE DISCLOSURE

According to an aspect of the disclosure, a system to prepare histological slides includes one or more processors. The system can include a patch generator that is executed by the one or more processors to generate a plurality of patches from a digital image. Each patch of the plurality of patches can include a plurality of pixels. The system can include a feature extractor to calculate, for each patch of the plurality of patches, values according to one or more sharpness metrics. The system can include a patch classifier to determine a blur score for each patch of the plurality of patches. The blur score of the patch can be determined using the calculated values of the one or more sharpness metrics of the patch. The patch classifier can generate a blur map based on the blur score for each patch of the plurality of patches. The blur map can include, for each patch of the plurality of patches, a parameter value between a first threshold value and a second threshold value that is based on the blur score of the patch.

In some implementations, the patch classifier can determine the blur score for each of the plurality of patches with one of a random forest regression algorithm or a logistic regression algorithm. In some implementations, the patch classifier can determine the blur score for each of the plurality of patches with a residual neural network.

In some implementations, the system can include a background detector to discard patches that includes background data. In some implementations, a portion of a first of the plurality of patches can overlap a portion of a second of the plurality of patches.

In some implementations, the feature extractor can calculate a plurality of values for each patch in the group of the plurality of patches. The one or more sharpness metrics can include at least one pixel intensity-based feature, gradient-based feature, transform-based feature, and perceptual-based feature. The one or more sharpness metrics can include a variance metric, a range histogram metric, an entropy histogram metric, a Mason and Green's histogram metric, a Mendelsohn and Mayall's histogram metric, a gradient metric, a sum of modified laplacian metric, a Tenengrad metric, a blur metric in the frequency domain, a DCT blur metric, a Haar wavelet transform metric, a Marziliano metric, and a cumulative probability of blur detection metric.

In some implementations, the system can include a background detector to convert the image to a grayscale image. The patch classifier can flag each patch in the group of the plurality of patches having the blur score above a predetermined threshold and can generate the blur map based on the flagged patches.

According to an aspect of the disclosure, a method for detecting a quantity of blur in images can include generating, by a blur detector including one or more processors, a plurality of patches from a digital image. Each patch of the plurality of patches can include a plurality of pixels. The method can include calculating, by the blur detector, for each patch of the plurality of patches, values according to one or more sharpness metrics. The method can include determining, by the blur detector, a blur score for each patch of the plurality of patches. The blur score of the patch can be determined using the calculated values of the one or more sharpness metrics of the patch. The method can include generating, by the blur detector, a blur map based on the blur score for each patch of the plurality of patches. The blur map can include, for each patch of the plurality of patches, a parameter value between a first threshold value and a second threshold value that is based on the blur score of the patch.

In some implementations, the method can include determining, by the blur detector, the blur score for each of the plurality of patches with one of a random forest regression algorithm or a logistic regression algorithm. In some implementations, the blur score for each of the plurality of patches can be calculated with a residual neural network.

In some implementations, the method can include discarding, by the blur detector, a patch that comprises background data. A portion of a first of the plurality of patches can overlap a portion of a second of the plurality of patches.

The method can include calculating, by the blur detector, a plurality of values for each patch in the group of the plurality of patches. The one or more sharpness metrics can include at least one pixel intensity-based feature, gradient-based feature, transform-based feature, or perceptual-based feature. The one or more sharpness metrics can include a variance metric, a range histogram metric, an entropy histogram metric, a Mason and Green's histogram metric, a Mendelsohn and Mayall's histogram metric, a gradient metric, a sum of modified laplacian metric, a Tenengrad metric, a blur metric in the frequency domain, a DCT blur metric, a Haar wavelet transform metric, a Marziliano metric, or a cumulative probability of blur detection metric.

The method can include converting the image to a grayscale image. The method can include flagging, by the blur detector, each patch in the group of the plurality of patches having the blur score above a predetermined threshold, and generating, by the blur detector, the blur map based on the flagged patches.

DETAILED DESCRIPTION

Section B describes embodiments of systems and methods to detect blur and generate blur maps.

Section C describes embodiments of system and method to determine saliency of tissue.

A. Computing and Network Environment

Prior to discussing specific embodiments of the present solution, it may be helpful to describe aspects of the operating environment as well as associated system components (e.g., hardware elements) in connection with the methods and systems described herein. Referring toFIG. 1A, an embodiment of a network environment is depicted. In brief overview, the network environment includes one or more clients102a-102n(also generally referred to as local machine(s)102, client(s)102, client node(s)102, client machine(s)102, client computer(s)102, client device(s)102, endpoint(s)102, or endpoint node(s)102) in communication with one or more servers106a-106n(also generally referred to as server(s)106, node106, or remote machine(s)106) via one or more networks104. In some embodiments, a client102has the capacity to function as both a client node seeking access to resources provided by a server and as a server providing access to hosted resources for other clients102a-102n.

Although,FIG. 1Ashows a network104between the clients102and the servers106, the clients102and the servers106may be on the same network104. In some embodiments, there are multiple networks104between the clients102and the servers106. In one of these embodiments, a network104′ (not shown) may be a private network and a network104may be a public network. In another of these embodiments, a network104may be a private network and a network104′ a public network. In still another of these embodiments, networks104and104′ may both be private networks.

The network104may be connected via wired or wireless links. Wired links may include Digital Subscriber Line (DSL), coaxial cable lines, or optical fiber lines. The wireless links may include BLUETOOTH, Wi-Fi, Worldwide Interoperability for Microwave Access (WiMAX), an infrared channel or satellite band. The wireless links may also include any cellular network standards used to communicate among mobile devices, including standards that qualify as 1G, 2G, 3G, or 4G. The network standards may qualify as one or more generation of mobile telecommunication standards by fulfilling a specification or standards such as the specifications maintained by International Telecommunication Union. The 3G standards, for example, may correspond to the International Mobile Telecommunications-2000 (IMT-2000) specification, and the 4G standards may correspond to the International Mobile Telecommunications Advanced (IMT-Advanced) specification. Examples of cellular network standards include AMPS, GSM, GPRS, UMTS, LTE, LTE Advanced, Mobile WiMAX, and WiMAX-Advanced. Cellular network standards may use various channel access methods e.g. FDMA, TDMA, CDMA, or SDMA. In some embodiments, different types of data may be transmitted via different links and standards. In other embodiments, the same types of data may be transmitted via different links and standards.

The network104may be any type and/or form of network. The geographical scope of the network104may vary widely and the network104can be a body area network (BAN), a personal area network (PAN), a local-area network (LAN), e.g. Intranet, a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The topology of the network104may be of any form and may include, e.g., any of the following: point-to-point, bus, star, ring, mesh, or tree. The network104may be an overlay network which is virtual and sits on top of one or more layers of other networks104′. The network104may be of any such network topology as known to those ordinarily skilled in the art capable of supporting the operations described herein. The network104may utilize different techniques and layers or stacks of protocols, including, e.g., the Ethernet protocol, the internet protocol suite (TCP/IP), the ATM (Asynchronous Transfer Mode) technique, the SONET (Synchronous Optical Networking) protocol, or the SDH (Synchronous Digital Hierarchy) protocol. The TCP/IP internet protocol suite may include application layer, transport layer, internet layer (including, e.g., IPv6), or the link layer. The network104may be a type of a broadcast network, a telecommunications network, a data communication network, or a computer network.

In some embodiments, the system may include multiple, logically-grouped servers106. In one of these embodiments, the logical group of servers may be referred to as a server farm38(not shown) or a machine farm38. In another of these embodiments, the servers106may be geographically dispersed. In other embodiments, a machine farm38may be administered as a single entity. In still other embodiments, the machine farm38includes a plurality of machine farms38. The servers106within each machine farm38can be heterogeneous—one or more of the servers106or machines106can operate according to one type of operating system platform (e.g., WINDOWS NT, manufactured by Microsoft Corp. of Redmond, Wash.), while one or more of the other servers106can operate on according to another type of operating system platform (e.g., Unix, Linux, or Mac OS X).

In one embodiment, servers106in the machine farm38may be stored in high-density rack systems, along with associated storage systems, and located in an enterprise data center. In this embodiment, consolidating the servers106in this way may improve system manageability, data security, the physical security of the system, and system performance by locating servers106and high-performance storage systems on localized high-performance networks. Centralizing the servers106and storage systems and coupling them with advanced system management tools allows more efficient use of server resources.

The servers106of each machine farm38do not need to be physically proximate to another server106in the same machine farm38. Thus, the group of servers106logically grouped as a machine farm38may be interconnected using a wide-area network (WAN) connection or a metropolitan-area network (MAN) connection. For example, a machine farm38may include servers106physically located in different continents or different regions of a continent, country, state, city, campus, or room. Data transmission speeds between servers106in the machine farm38can be increased if the servers106are connected using a local-area network (LAN) connection or some form of direct connection. Additionally, a heterogeneous machine farm38may include one or more servers106operating according to a type of operating system, while one or more other servers106execute one or more types of hypervisors rather than operating systems. In these embodiments, hypervisors may be used to emulate virtual hardware, partition physical hardware, virtualize physical hardware, and execute virtual machines that provide access to computing environments, allowing multiple operating systems to run concurrently on a host computer. Native hypervisors may run directly on the host computer. Hypervisors may include VMware ESX/ESXi, manufactured by VMWare, Inc., of Palo Alto, Calif.; the Xen hypervisor, an open source product whose development is overseen by Citrix Systems, Inc.; the HYPER-V hypervisors provided by Microsoft or others. Hosted hypervisors may run within an operating system on a second software level. Examples of hosted hypervisors may include VMware Workstation and VIRTUALBOX.

Management of the machine farm38may be de-centralized. For example, one or more servers106may comprise components, subsystems and modules to support one or more management services for the machine farm38. In one of these embodiments, one or more servers106provide functionality for management of dynamic data, including techniques for handling failover, data replication, and increasing the robustness of the machine farm38. Each server106may communicate with a persistent store and, in some embodiments, with a dynamic store.

Server106may be a file server, application server, web server, proxy server, appliance, network appliance, gateway, gateway server, virtualization server, deployment server, SSL VPN server, or firewall. In one embodiment, the server106may be referred to as a remote machine or a node. In another embodiment, a plurality of nodes290may be in the path between any two communicating servers.

Referring toFIG. 1B, a cloud computing environment is depicted. A cloud computing environment may provide client102with one or more resources provided by a network environment. The cloud computing environment may include one or more clients102a-102n, in communication with the cloud108over one or more networks104. Clients102may include, e.g., thick clients, thin clients, and zero clients. A thick client may provide at least some functionality even when disconnected from the cloud108or servers106. A thin client or a zero client may depend on the connection to the cloud108or server106to provide functionality. A zero client may depend on the cloud108or other networks104or servers106to retrieve operating system data for the client device. The cloud108may include back end platforms, e.g., servers106, storage, server farms or data centers.

The cloud108may be public, private, or hybrid. Public clouds may include public servers106that are maintained by third parties to the clients102or the owners of the clients. The servers106may be located off-site in remote geographical locations as disclosed above or otherwise. Public clouds may be connected to the servers106over a public network. Private clouds may include private servers106that are physically maintained by clients102or owners of clients. Private clouds may be connected to the servers106over a private network104. Hybrid clouds108may include both the private and public networks104and servers106.

Clients102may access IaaS resources with one or more IaaS standards, including, e.g., Amazon Elastic Compute Cloud (EC2), Open Cloud Computing Interface (OCCI), Cloud Infrastructure Management Interface (CIMI), or OpenStack standards. Some IaaS standards may allow clients access to resources over HTTP, and may use Representational State Transfer (REST) protocol or Simple Object Access Protocol (SOAP). Clients102may access PaaS resources with different PaaS interfaces. Some PaaS interfaces use HTTP packages, standard Java APIs, JavaMail API, Java Data Objects (JDO), Java Persistence API (JPA), Python APIs, web integration APIs for different programming languages including, e.g., Rack for Ruby, WSGI for Python, or PSGI for Perl, or other APIs that may be built on REST, HTTP, XML, or other protocols. Clients102may access SaaS resources through the use of web-based user interfaces, provided by a web browser (e.g. GOOGLE CHROME, Microsoft INTERNET EXPLORER, or Mozilla Firefox provided by Mozilla Foundation of Mountain View, Calif.). Clients102may also access SaaS resources through smartphone or tablet applications, including, e.g., Salesforce Sales Cloud, or Google Drive app. Clients102may also access SaaS resources through the client operating system, including, e.g., Windows file system for DROPBOX.

The client102and server106may be deployed as and/or executed on any type and form of computing device, e.g. a computer, network device or appliance capable of communicating on any type and form of network and performing the operations described herein.FIGS. 1C and 1Ddepict block diagrams of a computing device100useful for practicing an embodiment of the client102or a server106. As shown inFIGS. 1C and 1D, each computing device100includes a central processing unit121, and a main memory unit122. As shown inFIG. 1C, a computing device100may include a storage device128, an installation device116, a network interface118, an I/O controller123, display devices124a-124n, a keyboard126and a pointing device127, e.g. a mouse. The storage device128may include, without limitation, an operating system, software, and a software of blur detector120and saliency detector414. As shown inFIG. 1D, each computing device100may also include additional optional elements, e.g. a memory port103, a bridge170, one or more input/output devices130a-130n(generally referred to using reference numeral130), and a cache memory140in communication with the central processing unit121.

The central processing unit121is any logic circuitry that responds to and processes instructions fetched from the main memory unit122. In many embodiments, the central processing unit121is provided by a microprocessor unit, e.g.: those manufactured by Intel Corporation of Mountain View, Calif.; those manufactured by Motorola Corporation of Schaumburg, Ill.; the ARM processor and TEGRA system on a chip (SoC) manufactured by Nvidia of Santa Clara, Calif.; the POWER7 processor, those manufactured by International Business Machines of White Plains, N.Y.; or those manufactured by Advanced Micro Devices of Sunnyvale, Calif. The computing device100may be based on any of these processors, or any other processor capable of operating as described herein. The central processing unit121may utilize instruction level parallelism, thread level parallelism, different levels of cache, and multi-core processors. A multi-core processor may include two or more processing units on a single computing component. Examples of a multi-core processors include the AMD PHENOM IIX2, INTEL CORE i5 and INTEL CORE i7.

Main memory unit122may include one or more memory chips capable of storing data and allowing any storage location to be directly accessed by the microprocessor121. Main memory unit122may be volatile and faster than storage128memory. Main memory units122may be Dynamic random-access memory (DRAM) or any variants, including static random-access memory (SRAM), Burst SRAM or SynchBurst SRAM (BSRAM), Fast Page Mode DRAM (FPM DRAM), Enhanced DRAM (EDRAM), Extended Data Output RAM (EDO RAM), Extended Data Output DRAM (EDO DRAM), Burst Extended Data Output DRAM (BEDO DRAM), Single Data Rate Synchronous DRAM (SDR SDRAM), Double Data Rate SDRAM (DDR SDRAM), Direct Rambus DRAM (DRDRAM), or Extreme Data Rate DRAM (XDR DRAM). In some embodiments, the main memory122or the storage128may be non-volatile; e.g., non-volatile read access memory (NVRAM), flash memory non-volatile static RAM (nvSRAM), Ferroelectric RAM (FeRAM), Magnetoresistive RAM (MRAM), Phase-change memory (PRAM), conductive-bridging RAM (CBRAM), Silicon-Oxide-Nitride-Oxide-Silicon (SONOS), Resistive RAM (RRAM), Racetrack, Nano-RAM (NRAM), or Millipede memory. The main memory122may be based on any of the above described memory chips, or any other available memory chips capable of operating as described herein. In the embodiment shown inFIG. 1C, the processor121communicates with main memory122via a system bus150(described in more detail below).FIG. 1Ddepicts an embodiment of a computing device100in which the processor communicates directly with main memory122via a memory port103. For example, inFIG. 1Dthe main memory122may be DRDRAM.

Devices130a-130nmay include a combination of multiple input or output devices, including, e.g., Microsoft KINECT, Nintendo Wiimote for the WIT, Nintendo WII U GAMEPAD, or Apple IPHONE. Some devices130a-130nallow gesture recognition inputs through combining some of the inputs and outputs. Some devices130a-130nprovides for facial recognition which may be utilized as an input for different purposes including authentication and other commands. Some devices130a-130nprovides for voice recognition and inputs, including, e.g., Microsoft KINECT, SIRI for IPHONE by Apple, Google Now or Google Voice Search.

Additional devices130a-130nhave both input and output capabilities, including, e.g., haptic feedback devices, touchscreen displays, or multi-touch displays. Touchscreen, multi-touch displays, touchpads, touch mice, or other touch sensing devices may use different technologies to sense touch, including, e.g., capacitive, surface capacitive, projected capacitive touch (PCT), in-cell capacitive, resistive, infrared, waveguide, dispersive signal touch (DST), in-cell optical, surface acoustic wave (SAW), bending wave touch (BWT), or force-based sensing technologies. Some multi-touch devices may allow two or more contact points with the surface, allowing advanced functionality including, e.g., pinch, spread, rotate, scroll, or other gestures. Some touchscreen devices, including, e.g., Microsoft PIXELSENSE or Multi-Touch Collaboration Wall, may have larger surfaces, such as on a table-top or on a wall, and may also interact with other electronic devices. Some I/O devices130a-130n, display devices124a-124nor group of devices may be augment reality devices. The I/O devices may be controlled by an I/O controller123as shown inFIG. 1C. The I/O controller may control one or more I/O devices, such as, e.g., a keyboard126and a pointing device127, e.g., a mouse or optical pen. Furthermore, an I/O device may also provide storage and/or an installation medium116for the computing device100. In still other embodiments, the computing device100may provide USB connections (not shown) to receive handheld USB storage devices. In further embodiments, an I/O device130may be a bridge between the system bus150and an external communication bus, e.g. a USB bus, a SCSI bus, a FireWire bus, an Ethernet bus, a Gigabit Ethernet bus, a Fibre Channel bus, or a Thunderbolt bus.

In some embodiments, display devices124a-124nmay be connected to I/O controller123. Display devices may include, e.g., liquid crystal displays (LCD), thin film transistor LCD (TFT-LCD), blue phase LCD, electronic papers (e-ink) displays, flexile displays, light emitting diode displays (LED), digital light processing (DLP) displays, liquid crystal on silicon (LCOS) displays, organic light-emitting diode (OLED) displays, active-matrix organic light-emitting diode (AMOLED) displays, liquid crystal laser displays, time-multiplexed optical shutter (TMOS) displays, or 3D displays. Examples of 3D displays may use, e.g. stereoscopy, polarization filters, active shutters, or autostereoscopic. Display devices124a-124nmay also be a head-mounted display (HMD). In some embodiments, display devices124a-124nor the corresponding I/O controllers123may be controlled through or have hardware support for OPENGL or DIRECTX API or other graphics libraries.

In some embodiments, the computing device100may include or connect to multiple display devices124a-124n, which each may be of the same or different type and/or form. As such, any of the I/O devices130a-130nand/or the I/O controller123may include any type and/or form of suitable hardware, software, or combination of hardware and software to support, enable or provide for the connection and use of multiple display devices124a-124nby the computing device100. For example, the computing device100may include any type and/or form of video adapter, video card, driver, and/or library to interface, communicate, connect or otherwise use the display devices124a-124n. In one embodiment, a video adapter may include multiple connectors to interface to multiple display devices124a-124n. In other embodiments, the computing device100may include multiple video adapters, with each video adapter connected to one or more of the display devices124a-124n. In some embodiments, any portion of the operating system of the computing device100may be configured for using multiple displays124a-124n. In other embodiments, one or more of the display devices124a-124nmay be provided by one or more other computing devices100aor100bconnected to the computing device100, via the network104. In some embodiments software may be designed and constructed to use another computer's display device as a second display device124afor the computing device100. For example, in one embodiment, an Apple iPad may connect to a computing device100and use the display of the device100as an additional display screen that may be used as an extended desktop. One ordinarily skilled in the art will recognize and appreciate the various ways and embodiments that a computing device100may be configured to have multiple display devices124a-124n.

Referring again toFIG. 1C, the computing device100may comprise a storage device128(e.g. one or more hard disk drives or redundant arrays of independent disks) for storing an operating system or other related software, and for storing application software programs such as any program related to the image classification system software120. Examples of storage device128include, e.g., hard disk drive (HDD); optical drive including CD drive, DVD drive, or BLU-RAY drive; solid-state drive (SSD); USB flash drive; or any other device suitable for storing data. Some storage devices may include multiple volatile and non-volatile memories, including, e.g., solid state hybrid drives that combine hard disks with solid state cache. Some storage device128may be non-volatile, mutable, or read-only. Some storage device128may be internal and connect to the computing device100via a bus150. Some storage device128may be external and connect to the computing device100via a I/O device130that provides an external bus. Some storage device128may connect to the computing device100via the network interface118over a network104, including, e.g., the Remote Disk for MACBOOK AIR by Apple. Some client devices100may not require a non-volatile storage device128and may be thin clients or zero clients102. Some storage device128may also be used as an installation device116, and may be suitable for installing software and programs. Additionally, the operating system and the software can be run from a bootable medium, for example, a bootable CD, e.g. KNOPPIX, a bootable CD for GNU/Linux that is available as a GNU/Linux distribution from knoppix.net.

Client device100may also install software or application from an application distribution platform. Examples of application distribution platforms include the App Store for iOS provided by Apple, Inc., the Mac App Store provided by Apple, Inc., GOOGLE PLAY for Android OS provided by Google Inc., Chrome Webstore for CHROME OS provided by Google Inc., and Amazon Appstore for Android OS and KINDLE FIRE provided by Amazon.com, Inc. An application distribution platform may facilitate installation of software on a client device102. An application distribution platform may include a repository of applications on a server106or a cloud108, which the clients102a-102nmay access over a network104. An application distribution platform may include application developed and provided by various developers. A user of a client device102may select, purchase and/or download an application via the application distribution platform.

Furthermore, the computing device100may include a network interface118to interface to the network104through a variety of connections including, but not limited to, standard telephone lines LAN or WAN links (e.g., 802.11, T1, T3, Gigabit Ethernet, Infiniband), broadband connections (e.g., ISDN, Frame Relay, ATM, Gigabit Ethernet, Ethernet-over-SONET, ADSL, VDSL, BPON, GPON, fiber optical including FiOS), wireless connections, or some combination of any or all of the above. Connections can be established using a variety of communication protocols (e.g., TCP/IP, Ethernet, ARCNET, SONET, SDH, Fiber Distributed Data Interface (FDDI), IEEE 802.11a/b/g/n/ac CDMA, GSM, WiMax and direct asynchronous connections).

In one embodiment, the computing device100communicates with other computing devices100′ via any type and/or form of gateway or tunneling protocol e.g. Secure Socket Layer (SSL) or Transport Layer Security (TLS), or the Citrix Gateway Protocol manufactured by Citrix Systems, Inc. of Ft. Lauderdale, Fla. The network interface118may comprise a built-in network adapter, network interface card, PCMCIA network card, EXPRESSCARD network card, card bus network adapter, wireless network adapter, USB network adapter, modem or any other device suitable for interfacing the computing device100to any type of network capable of communication and performing the operations described herein.

The computer system100can be any workstation, telephone, desktop computer, laptop or notebook computer, netbook, ULTRABOOK, tablet, server, handheld computer, mobile telephone, smartphone or other portable telecommunications device, media playing device, a gaming system, mobile computing device, or any other type and/or form of computing, telecommunications or media device that is capable of communication. The computer system100has sufficient processor power and memory capacity to perform the operations described herein. In some embodiments, the computing device100may have different processors, operating systems, and input devices consistent with the device. The Samsung GALAXY smartphones, e.g., operate under the control of Android operating system developed by Google, Inc. GALAXY smartphones receive input via a touch interface.

In some embodiments, the computing device100is a gaming system. For example, the computer system100may comprise a PLAYSTATION 3, or PERSONAL PLAYSTATION PORTABLE (PSP), or a PLAYSTATION VITA device manufactured by the Sony Corporation of Tokyo, Japan, a NINTENDO DS, NINTENDO 3DS, NINTENDO WII, or a NINTENDO WII U device manufactured by Nintendo Co., Ltd., of Kyoto, Japan, an XBOX 360 device manufactured by the Microsoft Corporation of Redmond, Wash.

In some embodiments, the computing device100is a tablet e.g. the IPAD line of devices by Apple; GALAXY TAB family of devices by Samsung; or KINDLE FIRE, by Amazon.com, Inc. of Seattle, Wash. In other embodiments, the computing device100is an eBook reader, e.g. the KINDLE family of devices by Amazon.com, or NOOK family of devices by Barnes & Noble, Inc. of New York City, N.Y.

In some embodiments, the communications device102includes a combination of devices, e.g. a smartphone combined with a digital audio player or portable media player. For example, one of these embodiments is a smartphone, e.g. the IPHONE family of smartphones manufactured by Apple, Inc.; a Samsung GALAXY family of smartphones manufactured by Samsung, Inc; or a Motorola DROID family of smartphones. In yet another embodiment, the communications device102is a laptop or desktop computer equipped with a web browser and a microphone and speaker system, e.g. a telephony headset. In these embodiments, the communications devices102are web-enabled and can receive and initiate phone calls. In some embodiments, a laptop or desktop computer is also equipped with a webcam or other video capture device that enables video chat and video call.

B. Systems and Methods to Detect Blur

The present disclosure describes systems and methods to detect blur in digital images. The solution can be incorporated into the quality control systems of pathology and other slide scanners or can be a stand-alone solution. The solution can identify scanned images that include blur and cause the scanner to automatically rescan the blurry image. The solution can also identify regions of the scanned image that include blur. The solution can generate blur maps for each of the scanned images that identify regions of the scanned image that include blur. A blur map can be similar to a heat map. The blur map can indicate a degree of blurring at various regions within an image. Subsequent workflows can use the blur maps during the analysis of the scanned images. For example, when classifying the content of the scanned image, automated classification systems can ignore regions of the scanned images that the blur maps indicate include an amount of blur that exceeds a predetermined threshold.

FIG. 2illustrates a block diagram of an example blur detection system200. The system200includes the blur detector120. The blur detector120includes a patch generator202, a background detector204, a feature extractor206, and a patch classifier208. The blur detector120includes a database210. The database210can store images212, patches214, and blur maps216. The images212can be generated from pathology slides218. The pathology slides218can be digitized with a scanner220. In some implementations, the blur detector120is a component of the scanner220. For example, the blur detector120can be a component of the scanner's quality control system.

In some implementations, the patch generator202, background detector204, feature extractor206, and the patch classifier208can each be implemented as a set of software instructions, computer code, or logic that performs the functionality of each of these components as described further below. In some implementations, these components may instead by implemented by hardware, for example, using one or more field programmable gate arrays (FPGAs) and/or one or more application-specific integrated circuits (ASICs). In some implementations, these components can be implemented as a combination of hardware and software.

The blur detector120can receive digitized pathology slides218at an interface. The pathology slides218can be microscope slides that include excised tissue samples. The tissue samples can be prepared and stained according to pathology workflows to enable a pathologist to diagnose based on the prepared pathology slides218. The pathology slides218can be digitized with the scanner220. The scanner220can be a high-throughput scanner that automatically scans prepared pathology slides218. For example, the scanner220can be an Aperio AT2 whole slide scanner (made available by Leica Biosystems) or a Nanozoomer C9600 virtual slide light microscope scanner (made available by HAMAMATSU).

The blur detector120can include a patch generator202. The patch generator202can be any script, file, program, application, set of instructions, or computer-executable code that is configured to enable a computing device on which the patch generator202is executed to divide an input image212into a plurality of patches214. The patch generator202can store the generated patches214in the database210. In general, a patch214can refer to a portion of an image212and includes a plurality of pixels. The patch generator202can divide each image212into multiple patches214. In some implementations, the patch generator202can divide an image212into a plurality of patches according to a grid overlay. For example, the patch generator202can divide an image212having a resolution of 1,000 pixels by 1,000 pixels into patches each representing an adjacent square portion of the image212with a resolution of 10 pixels by 10 pixels, thereby creating 10,000 patches for the image212.

The patch generator202can be configured to generate the patches214in different shapes or fashions. For example, the patch generator202can generate patches214that have rectangular or irregular shapes. The patch214may not all have a uniform size. For example, portions of the image212that include background data can be divided into larger patches214, and the portions of the image212that include tissue can be divided into smaller patches214. The patches214can overlap one another. For example, the patch generator202can use a sliding window to generate a first patch214that overlaps a second patch214by 50%. A portion of an image212may be included within more than one patch for that image212.

In some implementations, the patch generator202can also provide data augmentation or other pre-processing of the patches214. For example, the patch generator202can apply a color normalization, intensity normalization, or other image processing technique to some or all of the patches214. In some implementations, the patch generator202can convert the images212into black and white images212.

The blur detector120can include a background detector204. The background detector204can be any script, file, program, application, set of instructions, or computer-executable code that is configured to enable a computing device on which the background detector204is executed to detect background portions of the image212. The pathology slides218include sectioned tissue. The tissue can include voids or may not extend to the perimeter of the pathology slides218. When the patch generator202digitizes the pathology slides218, the voids or other open areas in the slide can be recorded as background. The background can be portions of the images212that do not include tissue image data. The background detector204can process each of the patches214and indicate whether the respective patch214contains substantially background image data or tissue image data. The background detector204can flag, tag, label, or otherwise identify or indicate a patch214that is found to include substantially only tissue image data. The background detector204can flag patches214that include more than 50%, 60%, 70%, or 80% tissue image data. When flagged, the blur detector120can further process the image212. The background detector204can discard patches214that include substantially background data.

The blur detector120can include a feature extractor206. The feature extractor206can be any script, file, program, application, set of instructions, or computer-executable code that is configured to enable a computing device on which the feature extractor206is executed to calculate feature values for each of the patches214according to one or more sharpness metrics. The feature extractor206can calculate feature values by applying different sharpness metrics to the image data included in each of the patches214. Each of the feature values can be a single value, a vector, or a multidimensional matrix. The feature extractor206can use sharpness metrics from a plurality of different categories. The sharpness metric categories can include pixel intensity-based metrics, gradient-based metrics, transform-based metrics, and perceptual-based metrics, or any combination thereof. The feature extractor206can use multiple sharpness metrics from each of the categories. For example, the feature extractor206can use a total of 13 sharpness metrics: 5 pixel intensity-based metrics, 3 gradient-based metrics, 3 transform-based metrics, and 2 perceptual-based metrics.

In some implementations, the sharpness metrics can include the pixel intensity-based metric: variance metric; the intensity-based metrics: range histogram metric, entropy histogram metric, Mason and Green's histogram metric, Mendelsohn and Mayall's histogram metric; the gradient-based metrics: gradient metric, sum of modified laplacian metric, Tenengrad metric; the transform-based metrics: blur metric in the frequency domain, DCT blur metric, Haar wavelet transform metric; the perceptual-based metrics: Marziliano metric, cumulative probability of blur detection metric; or any combination thereof.

In some implementations, the feature extractor206can generate between about 1 and about 30 feature values, between about 1 and about 20 feature values, between about 1 and about 15 feature values, between about 1 and about 10 feature values, between about 1 and about 7 feature values, or between about 1 and about 5 feature values for each patch214.

The blur detector120can include a patch classifier208. The patch classifier208can be any script, file, program, application, set of instructions, or computer-executable code, that is configured to enable a computing device on which the patch classifier208is executed to determine blur scores for the patches214and generate blur maps216.

The patch classifier208can classify each of the patches214as blurry or non-blurry. The patch classifier208can generate a blur score that indicates the level of blur in each of the patches214. The patch classifier208can classify the patches214as blurry or non-blurry and generate the blur score based on the above-described feature values generated by the feature extractor206. The patch classifier208can use a random forest regression algorithm, a logistic regression algorithm, or a residual neural network to generate a blur score for each of the patches214.

In some implementations, the random forest can include between about 10 and about 1,500 trees, between about 100 and about 1,300 trees, between about 300 and about 1300 trees, between about 700 and about 1,200 trees, or between about 900 and about 1,100 trees. The prediction error variability for the random forest can be estimated by repeating a cross-validation (CV) analysis multiple times. During each of the CV tests, a different one of the images212can be excluded.

In some implementations, the patch classifier208can include a residual neural network. The neural network can be between about 10 and about 20 layers deep, about 15 and about 20 layers deep, or between about 18 and 20 layers deep. In some implementations, the neural network can accept color or grayscale images212.

The patch classifier208can, for example using the random forest or neural network, generate a blur score for each of the patches214. The patch classifier208can generate the blur score based on each of the feature values that are generated by the feature extractor206. The blur score can indicate a level of blur present in each of the patches214.

The patch classifier208can generate a blur map based on the blur scores for each of the patches214. In some implementations, the patch classifier208can apply a threshold to the blur score of each patch214. If the blur score is above the threshold, the patch214can be flagged, tagged, labeled, or otherwise identified or indicated as containing blur. If the blur score is below the threshold, the patch214can be flagged, tagged, labeled, or otherwise identified, or indicated as not containing blur.

The patch classifier208can generate blur scores for each of the patches214from an image212. The patch classifier208can generate the blur map216from the blur scores. The blur map216can be a standalone image. In some implementations, the blur map216can be overlaid onto the image212. The blur map216can indicate which regions of the image212include blur. The regions including the blur may not be suitable for further automated or human analysis.

In some implementations, the patch classifier208can generate a blur score of the entire image212based on the blur scores for each of the patches214that make up the image212. If the blur score for the image212is below a predetermined threshold, the image212can be discarded. The blur score for the entire image212can be generated by averaging the blur score for each of the patches214. In some implementations, patches214containing substantially only background image data can be excluded from the calculation to generate an average blur score for the entire image212. In some implementations, a weighted average can be used to generate the blur score for the entire image212. The weighted average can weight patches214near the middle of the image212more heavily than patches214near the periphery of the image212. In some implementations, the blur score for the entire image212can be the mode of the patches' blur scores (e.g., which blur score occurred the most often). The blur score for the entire image212can be binary—indicating whether the image212is blurry or not blurry. The binary blur score for the entire image212can be generated by determining the number of patches that have a blur score above a predetermined threshold. If the number of patches214with a blur score above the threshold is greater than a majority of the patches214, the patch classifier208can classify the entire image212as being blurry (e.g., having a binary yes for the blur score). In some implementations, the entire image212can be marked as blurry if more than 20%, more than 30%, more than 40%, more than 50%, more than 60%, or more than 70% of the patches214have a blur score above the predetermined threshold. In some implementations, when the blur score for the entire image212is above the predetermined threshold or when the binary blur score indicates that the image212is blurry, the patch classifier208can generate a notification that the pathology slide218from which the image212was generated should be rescanned. In some implementations, the blur detector120can be a quality control component of the scanner220, and the patch classifier208can cause the scanner220to automatically rescan the pathology slide218.

In some implementations, the patch classifier208can be trained with a plurality of input images212that are generated to include a predetermined amount of blur. For example, a training data set can be generated from images212that substantially do not include blur. A Gaussian (or other) blur filter can be applied to different portions of the originally-sharp, input images212. The different portions of the input images212that are blurred can be varied in size, location, and amount of blur. The location and amount of blur applied to the portions can be recorded to be used in the training process. Generation of the training data set and training the model is also discussed further in the Examples section.

The blur detector120can include a database210. The database210can be any type of computer-readable storage device. The storage device can include magnetic-based drives, optical-based drives, or solid-state drives. The blur detector120can store the images212, patches214, and blur maps216in the database210.

FIG. 3illustrates a block diagram of an example method300to detect a quantity of blur in images. The method300can include receiving an image (BLOCK302). The method300can include generating a plurality of patches from the image (BLOCK304). The method300can include calculating feature values for each of the patches (BLOCK306). The method300can include determining a blur score for each of the patches (BLOCK308). The method300can include generating a blur map (BLOCK310).

As set forth above, the method300can include receiving an image (BLOCK302). The image212can be a digitized image of a pathology slide218. The pathology slide218can be digitized by a scanner220. The blur detector120that performs at least portions of the method300can be a component of the scanner220or can be a component of a separate device. In some implementations, the blur detector can convert the image212into a grayscale image. In some implementations, the blur detector generates multiple images from the input image212. The multiple images can include different portions of the image spectrum. For example, the input image212can be divided into three images with the first containing the red components, the second containing the green components, and the third containing the blue components. In some implementations, the blur detector can perform other image preprocessing. For example, the blur detector may remove background portions of the image212, remove noise from the image212, or may normalize the image's212color histogram.

The method300can include generating a plurality of patches from the received image (BLOCK304). Also referring toFIG. 2andFIGS. 4A-4D, the patch generator202can divide the image212into a plurality of patches214. The patches214include a plurality of pixels. In some implementations, the patches214are generated in a grid-fashion as illustrated inFIG. 4A. In these implementations, the patches214do not overlay one another. In other implementations, the patch generator202can generate the patches214with a sliding window such that a patch214can overlay one or more of neighboring patches214.FIGS. 4B-4Dillustrate enlarged views of patches214(b)-214(d) from the image212illustrated inFIG. 4A. As illustrated, the patches214(b)-214(d) have different levels of blur. The patch214(b) is sharp and substantially lacks blur, the patch214(c) includes a moderate amount of blur, and the patch214(d) includes a relatively high degree of blur.

Referring toFIG. 3, the method300can include calculating feature values for each of the plurality of patches (BLOCK306). A plurality of sharpness metrics can be applied to each of the patches214. For each patch214, the feature extractor206can generate different feature values based on the different sharpness metrics. The feature extractor206can generate different feature values using a sharpness metric from different sharpness categories, such as pixel intensity-based features, gradient-based features, transform-based features, or perceptual-based features. The sharpness metrics can include a variance metric, a range histogram metric, an entropy histogram metric, a Mason and Green's histogram metric, a Mendelsohn and Mayall's histogram metric, a gradient metric, a sum of modified laplacian metric, a Tenengrad metric, a blur metric in the frequency domain, a DCT blur metric, a Haar wavelet transform metric, a Marziliano metric, or a cumulative probability of blur detection metric.

The method300can include determining a blur score for each of the patches (BLOCK308). The patch classifier208can generate the blur score for each of the patches214based on the feature values that the feature extractor206calculates for each of the respective patches214. The patch classifier208can use the feature values as inputs in determining the blur score. The patch classifier208can use the feature values as inputs to a random forest classifier or a neural network classifier. The neural network classifier can be a residual neural network. In some implementations, the blur scores for each of the image's patches214can be normalized—for example, between 0 and 1, with the blurriest patch214being normalized to 1 and the sharpest patch214being normalized to 0.

In some implementations, the blur score for each patch214can be converted to a binary result—blurry or not blurry. Each patch's blur score can be compared against a predetermined threshold. The patches214having a blur score above the predetermined threshold can be flagged as blurry and the patches214having a blur score below the predetermined threshold can be flagged as not blurry.

The method300can include generating a blur map (BLOCK310). The blur map can be based on the blur score for each of the patches214. The blur map can assign a parameter value to each region of the blur map based on the blur score from the patch214generated from the respective region. The parameter value can be between a first and second threshold value. For example, the blur map's parameter values can be normalized between 0 and 255 for a grayscale image. In some implementations, the parameter value can be set to a first threshold value (e.g., 0) or a second threshold value (e.g., 1) based on whether the blur score for the respective patch214is above or below a predetermined threshold.

Also referring toFIGS. 5A-5F,FIG. 5Aillustrates an image212.FIG. 5Billustrates a blur map500for the image212illustrated inFIG. 5A.FIGS. 5C-5Fillustrate enlarged patches214generated from the image212illustrated inFIG. 5A.FIG. 5Cillustrates a patch214that includes a relatively sharp region of the image212illustrated inFIG. 5A. The patch214illustrated inFIG. 5Cis taken from region A.FIGS. 5D and 5Eare taken from regions B and C of the image212illustrated inFIG. 5A. Both the patches214include a relatively high level of blur.FIG. 5Fis taken from region D of the image212illustrated inFIG. 5A. The patch214includes a mixture of blurry and non-blurry regions.

For the blur map500illustrated inFIG. 5B, the patch classifier208compared each patch's blur score to a threshold. The blur scores above the threshold were assigned a relative blur score of 1 and appear as shaded regions502on the blur map500. The blur score below the threshold were assigned a relative blur score of 0 and appear as clear regions504on the blur map500.

FIG. 6illustrates a blur map500where the parameter values for each of the regions are scaled between a first and second threshold value. The parameter values are scaled between a first threshold value (e.g., 0) and a second threshold value (e.g., 2.5). The first threshold parameter value can correspond to the sharpest regions in the blur map500and the second threshold parameter value can correspond to the blurriest regions in the blur map500.

The blur detector120can be used to detect and quantify blur in any type of digital image. For example, the blur detector120can detect and quantify blur in satellite images. The satellite images can include blur that is induced by the change in topography of the imaged area. The blur detector can detect which regions of the satellite image are blurry because of the changes in terrain, mis-focusing of the satellite's cameras, artifacts in the image, or other causes of blur in the image. In another example, the blur detector120can detect blur in images used for facial recognition. The blur detector120can detect and classify an image as blurry and cause the facial recognition system to re-take an image of the person's face.

The blur detector120can also be used in other systems that automatically identify and classify objects within digital images. The blur maps generated by the blur detector120can be used by the identification and classification systems to indicate which regions of the input images should be ignored (or otherwise reduced in importance) when performing identification and classification tasks.

Examples

The below examples describe examples of using the system described herein to generate blur maps for input images. The examples were conducted with a system similar to the system200, illustrated inFIG. 2.

To test the system, data sets were first generated that included a tailored level of sharpness for quantification. The data sets were generated from 30 tissue microarray (TMA) spots from clear cell renal cell carcinoma (kidney cancer) patients, and 159 whole slide images (WSIs) of prostate cancer. The prostate and kidney slides were scanned on an Aperio AT2 whole slide scanner (Leica Biosystems), whereas hippocampus slides were scanned on a Nanozoomer C9600 virtual slide light microscope scanner (HAMAMATSU). All slides were subsequently anonymized to protect patient privacy. To guarantee broad applicability of the final prediction models not only in terms of instrumentation (e.g. using different scanners), the three sets were processed with different immunohistochemical staining. The prostate slides were stained with H&E, the hippocampus slides with SDF-1, and the kidney slides with TOM20. All the slides were manually inspected to be completely free of blurred regions.

Squared grayscaled patches214of 64, 128, 256 and 512 pixels (without overlap) were extracted from the slides. The kidney samples originating from TMAs included wide areas of white background. Therefore, a thresholding approach (t=230 on grayscaled patches214) was used to exclude patches214with too much background data. The data set include artificially blurred patches214that were generated using a Gaussian filter simulating out of focus blur. By increasing the standard deviation parameter of the Gaussian filter, increasing levels of blurriness were obtained. The level of blur for each slide was also scored from 0 to 5 by a human grader.

The patch classifier208generated feature values for each of the patches214. The blur score was generated for each patch214with a random forest that included 1,000 trees trained with 13, 10, and 6 feature values. In another test, the patch classifier208generates the feature values using a residual neural network that included 18 layers of neurons. Training for the neural network was performed in parallel on four Nvidia TitanX GPUs for 300 epochs with hyper-parameters set as follows: batch size=1024, learning rate=0.1 (multiplied by 1/10 every 30 epochs), momentum=0.1. For each epoch, the training was done on the training set and validation error calculated for the validation set. After 300 epochs, the best performing model on the validation set was chosen.

In one implementation, the patch classifier208was trained on the tissue types separately. The patch classifier208was trained on kidney, prostate, and hippocampus images212. Then classification experiments were performed on single tissue data sets (e.g. prostate, hippocampus, and kidney data sets). The blur detection was highly accurate for all data sets with errors of 2%, 1.5%, and 0.3% for prostate, kidney, and hippocampus data sets, respectively, for patches214of 64×64 pixels. The error was 0.5%, 0.2%, and 0.5% for patches214of 512×512 pixels. It was observed that the size of the patches214can influence accuracy with blur detection on bigger patches214being more accurate than on smaller patches214as it can be seen inFIG. 7.FIG. 7illustrates, for each of the tissue data sets the relationship between patch size and error rate.

The feature importance analysis determined that, in the case of intra-set tasks, only 2 or 3 metrics would be enough to reach minimum prediction error, with perceptual metrics and cosine transform metric being the most important. Going one step further, all data sets were combined to allow for a more general classifier able to predict blurriness independently from the tissue type, at least within the tissues in our data set. Kidney samples with mitochondria staining were harder to predict with an error of around 3% mostly due to false positives (sharp patches214that were predicted blurred), while prostate and hippocampus had instead errors of 0.5% and 0.2% respectively, as shown inFIG. 8. The feature importance analysis highlighted how, in this case, more metrics can be used to discern blurriness when image content has higher variance.

A regressor was then trained on all 3 data sets to also predict the level of blurriness of the patches214. The results included a RMSD close to 0.012 and a Spearman correlation coefficient larger than 0.98. As illustrated inFIG. 9, the dispersion of the predictions is shown and a very flat distribution centered around the expected values can be observed, underlying the accuracy of the regression. The feature selection was performed minimizing the mean squared error (MSE) and it was in accord with the results of the classification task. The bar chart illustrated inFIG. 9illustrates how the metrics affect the classification.

The classification experiments were also performed where the patch classifier208used a neural network. The residual neural network (ResNet) converged to 0.03% error on the validation set after approximately 50 epochs. The best model showed 99.95% accuracy on the test set across all classes. The result was comparable to the random forest approach, which achieved an accuracy of 99.39%. The logistic regression showed an accuracy of 94%, pointing to the fact that the non-linearity introduced by the random forest or the neural network, can be important for this task. In addition, by using a reduced set of features only about 0.29% 0.66% accuracy for logistic regression and random forest, respectively, was lost. For mixed data sets, the validation error converged to 0.29% after roughly 100 epochs. For the mixed data sets, the accuracy was about 99.74%, outperforming the random forest approach with an accuracy 97.43%. The logistic regression had an accuracy of 87%. Using the reduced set of features, the accuracy drops by 4.18% and 1.08% for the logistic regression and random forest, respectively. A summary of these results is also presented in the below table.

The system described herein can also be used to determine the level of blur present in an image212.FIG. 10illustrates the determined blur score for a plurality of data sets. The data sets included a Gaussian blur of 0, 0.8, 1.2, 1.6, 2, and 2.4. The blur was detected with a ResNet classifier, a random forest classifier, and a random forest classifier using a reduced set of input sharpness metrics. As illustrated inFIG. 10, all the models were able to discern the different levels of blur with fair accuracy. The ResNet converged after roughly 30 epochs. Prediction on the test set using the best model gave an MSE of 0.018, whereas the two random forest approaches resulted in a MSE of 0.004 and 0.005 when using all features or a reduced set, respectively.

C. Systems and Method to Determine Slide Saliency

Most digital slides are annotated (or otherwise classified) at the whole slide level and not at the image (or sub-slide) level. The manual labeling of the images is prohibitive, requiring pathologists with decades of training and outstanding clinical service responsibilities. The present solution can generate annotations in a nonintrusive manner during a pathologist's routine clinical work. The solution can, after routine scanning of the whole slide, register the video frames of the pathologist's field of view to a digital version of the slide. The solution can detect the motion of the pathologist's view and measure the observing (or dwell) time to generate a spatial and temporal saliency heat map of the whole slide. The annotated slides and heat maps can be used to train a neural network to annotate and identify regions of important in future input slides. The solution also includes a convolutional neural network that detects diagnosis-relevant salient regions for new input slides and identifies the salient regions of the input slide.

FIG. 11illustrates a system400to detect the saliency of images. The system400includes a saliency detector414. The saliency detector414and the blur detector120can be part of the same device or different devices. The saliency detector414can include a patch generator202. The saliency detector414can include a motion detector402and a classifier404. The saliency detector414can include a heat map generator406that generates the heat maps408. The system400can include a microscope410that views pathology slides218. Via the microscope410, a camera412captures the images212of the pathology slide218. The saliency detector414can receive the images212from the camera412and store the images212in the database210.

The system400can include a camera412. The camera412can be a video camera or a still camera. The camera412can be a Panasonic Lumix DMC-FH10 camera with a 16.1 megapixel charge-coupled device (“CCD”), capable of 720p motion JPEG video at 30 frames per second. The camera412can be mounted on a second head of the microscope410. The microscope410can have an objective lens magnification of 4×, 10×, 20×, 40×, and 100×. The microscope410can have a magnification of 10×. The system400can include an additional camera412that can be a scanner220that creates a digital image of the slide. As described below, the video stream generated by the camera412can be registered with the digital image of the slide.

The saliency detector414can include a patch generator202. As described above, the patch generator202can generate a plurality of patches214from the images212that the saliency detector414receives. The patch generator202can store the patches214into the database210. The patches214can overlap neighboring patches214by between about 0% and about 80%, between about 0% and about 60%, or between about 0% and about 40%.

The motion detector402can detect movement of the pathology slides218along the microscope's stage to determine which portion of the pathology slide218(and which patch214) the viewer is viewing. In some implementations, the movement can be detected directly by the movement of the microscope stage.

In some implementations, the motion detector402can use computer vision to detect movement between frames of the video stream received from the camera412. For example, the motion detector402can detect the movement with OpenCV. In some implementations, the motion detector402determines a relative motion. In other implementations, the pathology slide's initial position can be registered with the motion detector402to enable the motion detector402to determine the absolute motion of the pathology slide218.

The motion detector402, running a computer vision script or library (e.g., OpenCV) can calculate a rigid body transformation between the interest points in a video frame and a patch214in order to calculate the distance in pixels that the video frame can be off-center from the patch214. The least off-center patch214can be selected as the best registration because the pathologist's field of view (“FOV”) can be in approximately the same place in this video frame and patch214.

During inspection of the pathology slide218by a pathologist, the pathologist can switch the objective lens magnification. The motion detector402can detect the lens change automatically when the field of view bounding box of nonblack pixels changes size. The motion detector402can also detect a switch in the object lens magnification by detecting a change in the pixel density of the image212(or portion thereof) analyzed by the motion detector402.

As the pathologist examines the pathology slide218, the pathologist can move the pathology slide218around the stage of the microscope410and view different regions of the pathology slide218. Based on the movement of the pathology slide218, the motion detector402can determine which of the patches214corresponding to the image212of the pathology slide218, the pathologist is viewing. The motion detector402can maintain a data matrix that includes a location corresponding to each of the patches214in the image212. The motion detector402can measure the amount of time the pathologist dwells (or views) a region of the pathology slide218that corresponds to one of the patches214(or a subregion of the patches214). At each patch214location in the data matrix, the motion detector402can store the total time that the pathologist viewed the respective patch214.

The saliency detector414can include a heat map generator406. The heat map generator406can generate heat maps408based on the amount of time the motion detector402determines the pathologist viewed each of the patches214of an image212.FIG. 12illustrates an example heat map408. The heat map408is divided along the plurality of patches214. In other implementations, the heat map408can be delineated along borders other than the patches214, such as subpatches smaller than the patches214. The heat map408can be divided into regions smaller or larger than the patches214. The heat map408can be overlaid on the image212for which the heat map408was generated. Each region (or patch214) can indicate the amount time the pathologist viewed the region. The amount of time can be indicated with a numerical value that indicates the time in seconds or by a color coding scheme.

As discussed above, the regions can be divided into regions smaller than the patches214.FIG. 12illustrates a region420that was subdivided into subpatches. The patch generator202can generate subpatches for each of the patches214. The subpatches can be generated when the pathologist zooms into a patch214to view the patch214at a higher resolution.

The saliency detector414can include a classifier404. The classifier404can, based on a comparison of the heat maps408to the images212, determine which regions (or patches214) of the images212are salient or otherwise relatively more important when classifying and annotating the pathology slides218.

The classifier404can include a deep learning model to determine a binary classification of the patches214as salient or non-salient. The classifier404can use 800×800 pixel patches214as input. The patches214can be labeled with the corresponding viewing (or dwell) time as determined by the motion detector402. The classifier404can be a neural network. The weights of the top layer of the neural network can be re-initialized after) pre-training. Two output neurons can be connected to the re-initialized, top layer. The classifier404can be trained on 800×800 pixel patches214for 10,000 iterations.

In some implementations, the classifier404can determine if each of the respective patches214are salient. The classifier404can generate a binary, yes or no, for each of the patches214. In some implementations, the classifier404can predict a time that a pathologist would dwell on each of the input patches214and then determine a saliency score. The saliency score of each patches214can be a function of the predicted dwell time. In some implementations, the classifier404can predict a dwell time that the classifier404can compare to a predetermined threshold. If the dwell time for a respective patch214is above the predetermined threshold, the patch214can marked as a salient patch. If the predicted dwell time is less than the predetermined threshold, the patch214can be marked as not salient. In some implementations, the predetermined threshold can be between about 0.1 and about 0.5 seconds or between about 0.1 and about 0.25 seconds. In some implementations, the classifier404can generate annotations for each of the regions based on the saliency score for the regions of the slide.

FIG. 13illustrates a block diagram of an example method450for determining the saliency of an input patch. The method450can include receiving an image (BLOCK451). The method450can include generating a plurality of patches from the image (BLOCK452). The method450can include determining a dwell time for each of the patches (BLOCK453). The method450can include training a classifier based on the dwell times (BLOCK454). The method450can include determining the saliency of an input patch (BLOCK455).

As set forth above, the method450can include receiving an image (BLOCK451). The image212can be digitized image of a pathology slide218. The pathology slide218can be digitized via a camera that captures a pathologist's view of the pathology slide218through a microscope410. The received image212can be a plurality of images, such as a video stream. In some implementations, the saliency detector414can receive an image212of substantially the whole pathology slide218and a video stream of the pathologist's view through of the pathology slide218through the microscope410.

The method300can include generating a plurality of patches214from the received image (BLOCK452). The patches214include a plurality of pixels. In some implementations, the patches214are generated in a grid-fashion. In these implementations, the patches214do not overlay one another. In other implementations, the patch generator202can generate the patches214with a sliding window such that a patch214can overlay one or more of neighboring patches214.

The method450can include determining a dwell time for each of the patches (BLOCK453). The motion detector402can receive an image212of the pathology slide218, which the patch generator202can generate into a plurality of patches214. The motion detector402can also receive a video stream from the camera412that captures the pathologist's view through the microscope410. The motion detector402can determine which the patches214the pathologist is viewing in the image212of the pathology slide218by registering the motion detected in the video stream to the image212of the pathology slide218. The motion detector402can calculate the dwell time by activating a different running timer for each of the patches214when the pathologist views the respective patch214. In some implementations, the motion detector402can know the frame rate of the input video stream from the camera412and can calculate the dwell time by counting the number of frames that includes each of the respective patches214.

The method450can include training the classifier with the dwell times for each of the patches (BLOCK454). The classifier404can receive the dwell times and associated patch214. The classifier404can include neural network classifier. Through the training process, the classifier404can learn to generate a predicted dwell given the image data in an input patch.

The method450can include determining the saliency of an input patch (BLOCK455). Once trained, the classifier404can generate a dwell time based on the image data in an input patch. The classifier404can convert the dwell time to a saliency value. In some implementations, the saliency value can be binary result that indicates whether the input patch214is salient or not. In some implementations, the saliency value can be a value that is a function of the dwell time. For example, a relatively longer dwell time can indicate that the patch214is relatively more salient than a patch214with a lower dwell time.

The saliency detector414can be used to generate heat maps for other digital or otherwise displayed content. For example, the saliency detector414can be used to design the layout of websites and other content systems. The saliency detector414can be trained by receiving gaze position from eye tracking systems that monitor the motion and gaze position of viewers. The system can then generate a heat map for the website. The system can register the heat map with the location of content on the website. The website and associated heat map can be used as training data for the system's classifier. Once trained, the system can generate predicted heat maps for new input websites. The system can suggest modifications to the layout of the input websites based on the predicted heat maps. For example, if the predicted heat maps indicate viewers of the website are not noticing content the website is promoting, the system can suggest a different layout to improve the likelihood the user will view the content.

The separation of various system components does not require separation in all implementations, and the described program components can be included in a single hardware or software product.

Where technical features in the drawings, detailed descriptions, or any claims are followed by reference signs, the reference signs have been included to increase the intelligibility of the drawings, detailed description, and claims. Accordingly, neither the reference signs nor their absence have any limiting effect on the scope of any claim elements.