Automated bounding box generation for objects in an image

One or more embodiments described herein include a computer-implemented method of determining a bounding box for an object in an image. The method includes determining a label for an object in a first image using a first algorithm, and generating a set of images based on the first image, by cropping the first image from a selected direction. The method further includes determining labels for each image in the set using the first algorithm, and removing images from the set such that the remaining images have a label matching the initial label. The method further includes determining a key image for the set, which is the smallest image from the set that has a confidence score exceeding a threshold. Further, the method includes determining a bounding box for the object in the first image based on a perimeter of a portion of the first image that overlaps the key image.

BACKGROUND

The present invention relates to computing technology, and particularly to techniques for automated bounding box generation for objects in an image. In one or more examples, such images with bounding boxes identified for objects in the images are used as training datasets.

Digital image based object detection has seen increased attention over the past few years. For example, object detection systems are currently being implemented in advanced driver assistance systems (ADAS), e-commerce applications, and various other areas. Conventional object detection methods usually involve two stages. First, in the detection stage, image regions that contain candidates of target objects are detected or localized. Then, in the recognition stage, such regions are further analyzed to recognize the specific content. However, these conventional object detection systems and methods generally require a large amount of training data, computing resources, have slow detection speeds, and can be inaccurate at times. Training data with bounding boxes for objects of interest is not easy to find and requires effort to generate.

SUMMARY

One or more embodiments described herein include a computer-implemented method of determining a bounding box for an object in an image. The method includes determining a label for an object in a first image using a first algorithm, and generating a set of images based on the first image, by cropping the first image from a selected direction. The method further includes determining labels for each image in the set using the first algorithm, and removing images from the set such that the remaining sections of the images have a label matching the initial label. The method further includes determining a key image for the set, which is the smallest image from the set that has a confidence score exceeding a threshold. Further, the method includes determining a bounding box for the object in the first image based on a perimeter of a portion of the first image that overlaps the key image.

According to one or more embodiments of the present invention, a system includes an image database with images that contain one or more objects, and labels identifying the one or more objects in the images. The system further includes processors configured to determine a bounding box for an object in a first image from the image database using a method that includes determining a label for an object in a first image using a first algorithm, and generating a set of images based on the first image, by cropping the first image from a selected direction. The method further includes determining labels for each image in the set using the first algorithm, and removing images from the set such that the remaining images have a label matching the initial label. The method further includes determining a key image for the set, which is the smallest image from the set that has a confidence score exceeding a threshold. Further, the method includes determining a bounding box for the object in the first image based on a perimeter of a portion of the first image that overlaps the key image.

According to one or more embodiments of the present invention a computer program product includes a computer readable storage medium having stored thereon program instructions executable by one or more processing devices to perform a method that includes determining a label for an object in a first image using a first algorithm, and generating a set of images based on the first image, by cropping the first image from a selected direction. The method further includes determining labels for each image in the set using the first algorithm, and removing images from the set such that the remaining images have a label matching the initial label. The method further includes determining a key image for the set, which is the smallest image from the set that has a confidence score exceeding a threshold. Further, the method includes determining a bounding box for the object in the first image based on a perimeter of a portion of the first image that overlaps the key image.

It is to be understood that the technical solutions are not limited in application to the details of construction and to the arrangements of the components set forth in the following description or illustrated in the drawings. The technical solutions are capable of embodiments in addition to those described and of being practiced and carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein, as well as the abstract, are for the purpose of description and should not be regarded as limiting. As such, those skilled in the art will appreciate that the conception upon which this disclosure is based may readily be utilized as a basis for the designing of other structures, methods, and systems for carrying out the several purposes of the presently described technical solutions.

DETAILED DESCRIPTION

It should be noted that the description herein refers to object detection in electronic images and as such references to “images” herein refer to electronic images, which may be stored in any electronic format, such as JPEG, PNG, and the like.

In computing technology, automatically detecting an object in an image and marking such a detected object in the image can be performed using artificial neural networks (ANN) or machine learning algorithms running on a programmable computer. For an ANN to operate as desired, the ANN has to be trained. In this case, for training an ANN to detect and mark objects in an image, the ANN has to be trained with images that already are associated with bounding boxes for objects in those images. The bounding boxes on such training images is usually done manually. Such manual labor to generate such training images can get expensive and is also slow.

Consequently, at this time, there are not a lot of training images available for ANNs to use and be trained for automatic object detection. This is a major technical challenge to large scale object detection in computing technology. For example, one of the largest databases of training images is the Microsoft Common Objects in COntext (MS COCO), which contains 91 common object categories with 82 of them having more than 5,000 labeled instances. It has a total of 328,000 images.

Accordingly, embodiments of the present invention provide techniques of using an existing ANN, which is trained to recognize a large number of different objects to help in automatically creating the bounding boxes for the detected objects in the training images. According to one or more embodiments of the present invention marking objects in the training images can be done relatively quickly and over a much larger data set, thus addressing the technical problem described herein.

For example, embodiments of the present invention can be used to generate bounding boxes on images available via image databases in which each image is marked with a label identifying one or more objects in the image. Such image databases include IMAGENET-22K with 14 million images, PLACES-2 with 8 million images, OPENIMAGES, and other such image databases.

FIG. 1depicts an existing system that is used for marking training images with bounding boxes. Here, a user105(human) uses a bounding box marking system110to manually create a training image set130from an image set120. The user creates the training image set130by manually marking a bounding box135on an object125in each image in the image set130. A corresponding image with the bounding box135is stored in the training image set130. It is understood that the depicted objects are examples and that in one or more embodiments of the present invention the image set130can contain a large number of images (thousands, millions etc.) and that each image can contain one or more objects125, which can be more complex than those depicted inFIG. 1. The training image set130thus created is used to train an object detection system140that can include an ANN. The object detection system140, based on the training, can automatically detect one or more objects in a query image150.

The bounding box marking system110and the object detection system140can be computing devices, such as computer servers, desktop computers, laptop computers, tablet computers, phones, or any other computing devices. The sets of images (120and130) can be accessed/stored on database systems.

FIG. 2illustrates an example system200. The system200represents a computing device that communicates via a network265. The system200includes hardware, such as electronic circuitry and can be any of the computing device used by one or more embodiments of the present invention.

The system200includes, among other components, a processor205, memory210coupled to a memory controller215, and one or more input devices245and/or output devices240, such as peripheral or control devices, that are communicatively coupled via a local I/O controller235. These devices240and245may include, for example, battery sensors, position sensors, indicator/identification lights and the like. Input devices such as a conventional keyboard250and mouse255may be coupled to the I/O controller235. The I/O controller235may be, for example, one or more buses or other wired or wireless connections, as are known in the art. The I/O controller235may have additional elements, which are omitted for simplicity, such as controllers, buffers (caches), drivers, repeaters, and receivers, to enable communications.

The processor205is a hardware device for executing hardware instructions or software, particularly those stored in memory210. The processor205may be a custom made or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the system200, a semiconductor based microprocessor (in the form of a microchip or chip set), a macroprocessor, or other device for executing instructions. The processor205includes a cache270, which may include, but is not limited to, an instruction cache to speed up executable instruction fetch, a data cache to speed up data fetch and store, and a translation lookaside buffer (TLB) used to speed up virtual-to-physical address translation for both executable instructions and data. The cache270may be organized as a hierarchy of more cache levels (L1, L2, and so on.).

The memory210may include one or combinations of volatile memory elements (for example, random access memory, RAM, such as DRAM, SRAM, SDRAM) and nonvolatile memory elements (for example, ROM, erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), tape, compact disc read only memory (CD-ROM), disk, diskette, cartridge, cassette or the like). Moreover, the memory210may incorporate electronic, magnetic, optical, or other types of storage media. Note that the memory210may have a distributed architecture, where various components are situated remote from one another but may be accessed by the processor205.

The instructions in memory210may include one or more separate programs, each of which comprises an ordered listing of executable instructions for implementing logical functions. In the example ofFIG. 2, the instructions in the memory210include a suitable operating system (OS)211. The operating system211essentially may control the execution of other computer programs and provides scheduling, input-output control, file and data management, memory management, and communication control and related services.

Additional data, including, for example, instructions for the processor205or other retrievable information, may be stored in storage220, which may be a storage device such as a hard disk drive or solid state drive. The stored instructions in memory210or in storage220may include those enabling the processor to execute one or more aspects of the systems and methods described herein.

The system200may further include a display controller225coupled to a user interface or display230. In some embodiments, the display230may be an LCD screen. In other embodiments, the display230may include a plurality of LED status lights. In some embodiments, the system200may further include a network interface260for coupling to a network265. The network265may be an IP-based network for communication between the system200and an external server, client and the like via a broadband connection. In an embodiment, the network265may be a satellite network. The network265transmits and receives data between the system200and external systems. In some embodiments, the network265may be a managed IP network administered by a service provider. The network265may be implemented in a wireless fashion, for example, using wireless protocols and technologies, such as WiFi, WiMax, satellite, or any other. The network265may also be a packet-switched network such as a local area network, wide area network, metropolitan area network, the Internet, or other similar type of network environment. The network265may be a fixed wireless network, a wireless local area network (LAN), a wireless wide area network (WAN) a personal area network (PAN), a virtual private network (VPN), intranet or other suitable network system and may include equipment for receiving and transmitting signals.

As described earlier, the existing system that is used for marking training images with bounding boxes requires substantial manual input, which increases cost and time required for generating the training image set130that is used for training the ANN of the object detection system140.

Accordingly, one or more embodiments of the present invention use a second ANN to automatically generate bounding boxes and generate the training image set130. The ANN can generate the bounding boxes as an extremely parallel task, and hence, can be accelerated by using more machines to execute the ANN. Further, the one or more embodiments of the present invention facilitate using the ANN for base objects as well as custom objects.

FIG. 3depicts a block diagram of a system for automatically generating the training image set according to one or more embodiments of the present invention. The depicted system300includes an automatic bounding box marking system310that uses a labeled image set310. The automatic bounding box marking system310can be a computing device. The labeled image set310includes multiple images and multiple labels325of the objects125included in the images. For example, as depicted, a first label325indicates that a first image includes object-1, a second label indicates that a second image includes object-2, a third label indicates that the second image also includes object-3, and so on.

The automatic bounding box marking system310generates the training image set130that includes the images from the labeled image set120, with each image including one or more bounding boxes135. A bounding box135marks the boundaries of a corresponding object125in the image. In a 2D image, the bounding box125includes four sides, each side marking a boundary of the object125in a particular orientation. For example, in case of the 2D image, the bounding box includes a top side, a left side, a right side, and a bottom side, each corresponding to the top, left, right, and bottom edges of the image, respectively. The bounding box125may be a rectangle, a square, or any other 2D enclosure. In one or more examples, the bounding box135can be stored as coordinates that indicate the four sides of the bounding box135. Any other technique can be used to store the bounding box135.

FIG. 4depicts a flowchart for a method of automatically generating the training image set according to one or more embodiments of the present invention. Here, automatic generation is performed without the user105marking the bounding boxes on the images. As depicted, the method includes accessing the images from the labeled image set320. The labeled image set130includes multiple images and labels325associated with the images to indicate the objects125in the images. For example, in the case of IMAGENET-22K, and PLACES-2 datasets the labeled image set320includes 14 million images and 8 million images, respectively.

Each image in the labeled image set320is passed through a first ANN to determine if the ANN can detect the object125corresponding to a label325that is associated with that image (410). If the object125can be detected by the first ANN (420), cropped copies of the image are created (430). Creating the cropped copies includes making n copies of the image (432) and cropping each copy of the image from a selected orientation by a predetermined amount (434). For example, say the selected orientation is right and the predetermined amount is i*(1/n)* 100% of the image, where i indicates which copy is being cropped (434). Here, n is a predetermined number of copies, and can be configured. For example, if n is 50, the first copy of the image (i=1) will be cropped 2% from the right, the 2nd copy (i=2) is cropped 4% and so on. Here, 2% indicates that 2% of the length of the image is cropped from the selected orientation, which in this example is right.

FIG. 5depicts the above example scenario visually. A labeled image510from the labeled image set320is selected for generating the bounding box. Here, the selected orientation for the cropping is from the right. After making n copies of the image510, each copy is cropped by a corresponding predetermined amount that depends on which copy is being cropped, each copy being cropped by a different amount. In a similar manner, the labeled image510can be cropped in other orientations, such as top, down, and left.

Referring toFIG. 4, each copy of the image, is passed through the object detection ANN to determine if the object125can be detected from the copy that matches with the label325that is associated with that image (440). In one or more examples, a confidence score is also generated by the object detection ANN, the confidence score indicating a level of confidence for the detected object. It is to be noted that the object detection ANN can be run on a graphics processing unit (GPU), central processing unit (CPU) or any other type of processing unit. On hardware (like GPUs), that facilitate parallel processing, the method can include passing multiple images in parallel for processing as described herein. Accordingly, the method described herein can be accelerated by processing multiple images concurrently in this manner.

Further, the method includes finding, from the cropped copies520, an image-copy with bounding box edge in the selected orientation (440). Selecting the image-copy includes, determining, from the cropped copies520, a discard set530of the cropped copies (442). The discard set530includes the cropped copies that result in a different label535than the label325that was associated with the image510. For the remaining cropped copies, (non-discard) the image-copy550is selected that has the largest value of i and which resulted in at least a predetermined threshold of confidence score when detecting the object125per the label325(444). The predetermined threshold can be an absolute value (e.g. 80% confidence), or a proportional value (90% of highest confidence value of the cropped copies). The selected image-copy550is retained for further processing and all other copies are discarded (446).

The method further includes determining if the image510has been processed in all orientations (top, right, bottom, left) (450). If not, a next orientation is selected and the retained selected image-copy550is processed again as described herein (460). The order in which the orientations are processed van vary from one example to another. Once all orientations have been processed (460), the resultant selected image-copy550denotes a minimal image where the object125can be identified with at least the predetermined threshold confidence and the edges of the object are a candidate bounding box135(470).

In one or more examples, the method is repeated with a different order of the orientations being selected. For example, in a first iteration the orientations may be selected from left to right or top to bottom, while in a second iteration, the orientations are selected in any other order. Each iteration can result in a different candidate bounding box. In case multiple candidate bounding boxes are generated, any of those candidate bounding boxes or a combination of the candidate bounding boxes can be considered as the bounding box135. For example, a combination of the candidate bounding boxes can be the maximal encompassing box from the candidate bounding boxes, which is used as the bounding box135.

One or more embodiments of the present invention facilitate determining a bounding box for an object in an image. A method for determining the bounding box includes determining a label for an object in a first image using a first neural network. Further, the method includes, for each of a right direction, left direction, top direction, and bottom direction, generating a set of images based on the first image, wherein each image of the set of images is generated by cropping the first image from a given direction. The method further includes determining labels for each image in the set of images using the first neural network. The method further includes removing images from the set of images such that each image remaining in the set of images has a label matching the label for the object in the first image. Further, the method includes determining a key image for the set of images, wherein the key image is the smallest image of the set of images that has a confidence score exceeding a threshold, and determining a bounding box for the object in the first image based on the perimeter of the portion of the first image that overlaps each key image for each direction.

Instead of generating the bounding boxes by operating multiple slices of the images concurrently, one or more embodiments of the present invention facilitate automatically determining a bounding box for an object in an image by repetitively slicing the image by cropping the image and determining if the cropped portion contains the object. A method for determining the bounding box includes determining a label for an object in an image. The method further includes generating a first cropped image by cropping the image by a predetermined amount along a first orientation (or direction). The method further includes determining that the first cropped image includes the object, and in which case, replacing the image with the first cropped image. Based on determining that the first cropped image does not include the object, determining a bounding box for the object in the image by using dimensions of the first copped image.

The bounding box is a first bounding box along the first orientation, and further, the method includes determining a second bounding box along a second orientation by cropping the image along the second orientation. In one or more examples, the final image from the iteration for the first orientation is used as the starting image for the second orientation. The first bounding box and the second bounding box are for the same object. In one or more examples, the first bounding box and the second bounding box are for different objects. As described herein, the label for the object is determined by a neural network, separate from a neural network that uses the generated bounding boxes as training data.

Accordingly, one or more embodiments of the present invention facilitate autonomous generation of bounding boxes for one or more image sets. Each of the bounding boxes encompasses an object in an image from an image set. The bounding boxes are generated autonomously using machine learning, such as using neural networks. The bounding boxes thus generated are used for creating training data for other machine learning systems, such as neural network systems for object detection for images. For example, using IMAGENET-22K, one or more embodiments of the present invention can generate bounding boxes for 15000 to 21000 different objects and labels and is orders of magnitude faster compared to existing solutions for generating such bounding boxes.

The one or more embodiments of the present invention accordingly provide an improvement to computer technology, for example in the case of neural network training systems that require images that have bounding boxes for objects in the images for training one or more neural networks. The one or more embodiments of the present invention described herein use 2D images as examples, however, the one or more embodiments of the present invention can be used for generating bounding boxes for objects in 3D images, or other types of images. Further, one or more embodiments of the present invention, can be used to determine bounding boxes for n-dimensional objects.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention.

In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s).