System and method for oriented bounding box tool defining an orientation of a tilted or rotated object

Systems and methods for an improved bounding box tool. The system can have processors configured to display a user interface on a display of a device, the user interface displaying image data. The processor can activate a virtual tool to define a bounding box. The processor can receive a first input data point at a first location relative to the image data. This can be defined by actuation of the input device. The processor can receive a movement input in a direction relative to the first location. The movement input can be defined by movement commands from the input device. The processor can receive a second input data point at a second location relative to the image data. The second input data point can be triggered by a second actuation of the input device. The processor can display, at the user interface, a graphical object representing the bounding box as an overlay of the image data. The bounding box can have corners defined by the first location and the second location. Edges (and angles thereof) of the bounding box can be defined by the direction of the movement input.

FIELD

The present disclosure generally relates to the field of artificial intelligence, computer vision, image processing, and interfaces.

INTRODUCTION

Embodiments described herein relate to training artificial intelligence systems for tasks using interface tools to define bounding boxes or volumes around objects or subjects of images. In order to train an artificial intelligence system for a particular task, such as a vision task, regions of images can be labelled by drawing bounding boxes (BB) around objects or subjects of images. In some cases, an oriented bounding box (OBB) is necessary to define an orientation for the object, but it can add complexities to the labeling task.

SUMMARY

In accordance with an aspect, there is provided a system for a bounding box tool. The system has a server having non-transitory computer readable storage medium with executable instructions for causing one or more processors to: display a user interface on a display of a device, the user interface displaying image data; activate a virtual tool to define a bounding box, the virtual tool controlled by commands received from an input device; display, at the user interface, an indicator for the virtual tool relative to the image data; receive a first input data point at a first location relative to the image data, the first input data point triggered by a first actuation of the input device; receive a movement input in a direction relative to the first location, the movement input defined by movement commands from the input device during the actuation of the input device and a release of the first actuation of the input device; receive a second input data point at a second location relative to the image data, the second input data point triggered by a second actuation of the input device; compute the bounding box in a bounding box format using the first input data point, the movement input, and the second input data point, wherein the bounding box format defines a first corner at the first location, an opposite corner at the second location, and an adjacent corner connected to the first corner by an edge at an angle defined by the movement input; automatically generate a new data file for storing, at a data store, the bounding box in the bounding box format; automatically generate a graphical object representing the bounding box using the new data file; display, at the user interface, the graphical object representing the bounding box as an overlay of the image data, the bounding box having corners defined by the first location and the second location, the bounding box having edges connecting the corners, the edges defined by the direction of the movement input.

In accordance with an aspect, there is provided a system for a bounding box tool. The system can have a server having non-transitory computer readable storage medium with executable instructions for causing one or more processors to: display a user interface on a display of a device, the user interface displaying image data; activate a virtual tool to define a bounding box, the virtual tool controlled by commands received from an input device; display, at the user interface, an indicator for the virtual tool relative to the image data; receive a first input data point at a first location relative to the image data, the first input data point triggered by a first actuation of the input device; receive a movement input in a direction relative to the first location, the movement input defined by movement commands from the input device during the actuation of the input device and a release of the first actuation of the input device; receive a second input data point at a second location relative to the image data, the second input data point triggered by a second actuation of the input device; display, at the user interface, a graphical object representing the bounding box as an overlay of the image data, the bounding box having corners defined by the first location and the second location, the bounding box having edges connecting the corners, the edges defined by the direction of the movement input.

In various further aspects, the disclosure provides corresponding systems and devices, and logic structures such as machine-executable coded instruction sets for implementing such systems, devices, and methods.

In this respect, before explaining at least one embodiment in detail, it is to be understood that the embodiments are not limited in application to the details of construction and to the arrangements of the components set forth in the following description or illustrated in the drawings. Also, it is to be understood that the phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting.

DETAILED DESCRIPTION

Embodiments of methods, systems, and apparatus are described through reference to the drawings.

Embodiments described herein relate to training artificial intelligence systems for tasks (e.g. vision tasks) using interface tools to define bounding boxes or volumes around objects or subjects of images. Training systems for vision tasks can involve labelling regions of images by drawing bounding boxes (BB) around objects or subjects of images. A BB can be a cuboid (in three dimensions) or a rectangle (in two dimensions) containing the object. That is, a BB is a volume that bounds or encapsulates one or more objects. An example vision task is object collision detection. Intersection of two BBs can indicate a collision of two objects, for example. The BB can be tight around the objects or subjects.

A BB can be aligned with the axes of the coordinate system (for the interface) and it can be referred to as an axis-aligned bounding box (AABB). The AABB can be a rectangular 4-sided box in two dimensions and 6-sided box in three dimensions categorized by having its faces oriented along the coordinate system. An AABB can be a rectangular box whose face normals are parallel to the axes of the coordinate system.

An oriented bounding box (OBB) is an arbitrarily oriented rectangular bound so that it uses an object's local coordinate system. In some cases, an OBB is necessary to define an orientation for the object such as if the object is tilted or rotated. For example,FIG. 1shows an interface with an image of a tilted TEXT object100to label. An OBB can add complexities to the labeling and vision task. For example, to detect collisions of objects the BBs for the object can be expressed within a common coordinate system. The objects may not have a common orientation and the OBBs may need to be converted for a particular vision task, for example. OBBs can provide models that are rotation-invariant or models that do not need to learn to detect objects that are oriented in multiple angles. For instance, OCR models can be simplified when they can assume that characters are laid out on an imaginary horizontal line in an image.

An example workflow to draw an OBB using an interface tool can be similar to drawing a rectangle, for example. An interface tool might not be well adapted to the labeling task, making it inefficient. When labeling tasks consist of drawing thousands of boxes, time efficiency becomes important. An example workflow can involve two steps. First, the user can actuate an input device (e.g. press down the mouse button) to trigger the interface tool to define one corner of the BB102. Next, the user can drag the input device and release the actuation to define the opposite corner of the BB102to define an AABB. Second, the user can actuate the input device to rotate the BB102to try to align it with the TEXT object100orientation, such as using a handle104of the interface (FIG. 2). The rotated BB102can be referred to as an OBB as it is oriented to the object's local coordinate system.

In order to end up with the correct BB102(e.g. correctly aligned to the orientation of the TEXT object102and correct horizontal and vertical dimensions to contain the TEXT object102), the user would have to guess at the first step the desired horizontal and vertical dimensions, as well as its horizontal and vertical center position, without any landmark. If BB102is not correct (which is common given the visual estimation required by this method) then the user must subsequently adjust each side, one by one. This can be inefficient.

Embodiments described herein provide an improved interface tool for drawing or defining a BB. For example, the improved method can enable drawing or defining an OBB using “one-shot” or user actuation of an input tool. The improved method can result in few or no subsequent adjustments.

FIG. 3shows an interface with a tilted TEXT object100to label using an improved virtual tool configured to define an OBB102according to some embodiments. The virtual tool can be controlled (e.g. actuated, moved) by an input device. The virtual tool can be used to capture input data for defining the OBB102. The input device can be a computer mouse, for example. The input device can be integrated with a touch sensitive display or surface, as another example. The user can interact with the input device using various gestures or movements to actuate the device. For example, the user can press down a mouse button (click) and drag or move the mouse to a location on the interface or image displayed thereon. As another example, the user can touch a touch sensitive display or surface and swipe his or her finder across the display or surface.

The workflow can involve two well-integrated steps. First, the user can actuate an input device (e.g. press down the mouse button) to trigger the interface tool to define one corner of the BB102. Then the user can drag the input tool and release the actuation of the input device to define the angle of one edge of the BB102. Second, the user can actuate the input device (e.g. click the mouse button) to define the opposite corner of the BB102to produce an OBB. The workflow can involve the following example gestures by the user can define the input data: click, drag, click; or touch, swipe, touch. The input device can have different hardware form factors to enable different gestures for the input data. For example, the input device can be integrated with a touch sensitive display device or surface. As another, example, the input device can be a computer mouse. The input device can have actuating buttons or gesture response. The input device can be dynamically adapted to different input tools to receive input data points. As another example, the input device can have natural motion gesture-based sensors for receiving input data. This can be in three dimensions, such as, remote gesture device, virtual reality device, depth camera, camera, stereo camera, and so on.

The example BB102around text100can be defined by a first input data point (for a first corner location), a movement input302of the virtual tool (for the edge angle), and a second input data point304(for the second corner location).

The interface can provide a virtual (drafting or bounding box) tool. The interface can receive a command for activation of the virtual tool to define the BB102and display the BB102as a graphical object of the interface. The interface can display an image (or more generally image data) and the virtual tool can move to different locations within the image to define a bounding box as an overlay of the image. That is, the bounding box can be defined using the virtual tool and displayed at the interface as a graphical object overlayed on the image. The location of the virtual tool within the interface can be controlled by the input device. The input device can also transmit commands to control, actuate, or manipulate the virtual tool. Movement of the input device can trigger corresponding movement of the virtual tool. Actuation of the input device can trigger a command for the virtual tool. The first actuation of the input device can be referred to as a first input data point at a first location. The drag or movement of the input device can be referred to as movement or direction input which can be defined relative to the first input data point. The second actuation of the input device can be referred to as a second input data point at a second location. The first input data point at the first location can define the first corner of the BB102. The movement input from the first location can define the edge angle of the BB102. The second input data point at the second location can define another (opposite) corner of the BB102. The other corner is opposite the first corner of the BB102. If the user selected the adjacent corner with the second input (click) by accident this can result in a BB102(e.g. OBB) that would collapse to a line. Since this is unlikely to be usable, embodiments described herein can disallow the second click if the box height is not greater than a threshold e.g. a few pixels and/or a percentage of the width, for example.

The method can also enable drawing or defining an AABB using the same technique.FIG. 4shows an interface with a tilted TEXT object100to label using the improved method to define an AABB. At402, the user actuates the input device (e.g. press down the mouse button) to trigger the interface (or virtual) tool to define one corner of the BB102. In this example, the user does not drag the input device and the default angle is computed as null. At404, the user can actuate the input device (e.g. click the mouse button) to define the opposite corner of the BB102to produce an AABB.

During the first step, while the user moves (e.g. drags, swipes) the input device to define the first edge, the interface can dynamically update to display a line between the corner and the location of the virtual tool (e.g. which can be displayed as a cursor), soon to become an edge of the BB. Then, until the second actuation of the input device (e.g. to define the second input data point), the suggested box can be displayed. The suggested box can be defined based on a current location of the virtual tool (e.g. the current position of a cursor before the actuation). The displayed BB can be set or frozen when the user actuates the input device to capture the second input data point and define the (second) corner.

The user can have the freedom to start by any of the four corners, and then to drag the input device to any of the two possible adjacent edges. In practice, the angle of the first edge drawn can be wrapped between −45° and 45°. The width and height can be exchanged if the angle has been wrapped by an odd number of quadrants.

Embodiments described herein provide an improved virtual tool for an interface to improve the workflow for a user to draw an OBB around a region of an image. The tool can also be used to define an AABB using the improved workflow to provide flexibility and avoid requiring the user to use different workflows for OBBs and AABBs. The OBB can be used for subsequent vision tasks (or other tasks). The OBB can be saved in different data formats. For example, an example format for an OBB is “angle-center-extent”, consisting of the angle, the center coordinates, and the extent (half-width and half-height). To extract the region of interest from an image, one can rotate the image around the OBB's center by the opposite of the OBB's angle, before cropping the image from pixel center-extent to pixel center+extent.

FIG. 8is a diagram of a system200for generating an interface with a virtual tool for BBs and an example physical environment.

The system200can include an I/O Unit102, a processor104, communication interface106, and data storage110. The processor104can execute instructions in memory108to implement aspects of processes described herein. The processor104can execute instructions in memory108to configure an interface controller220for generating and managing the interface for displaying image data222, a virtual tool224for defining BBs and generating bounding box data226, and other functions described herein. The system200may be software (e.g., code segments compiled into machine code), hardware, embedded firmware, or a combination of software and hardware, according to various embodiments.

In some embodiments, the system200can implement one or more vision tasks228using the image data222and bounding box data226. In some embodiments, the system200can connect to one or more vision applications230that can use the bounding box data226to define regions with the image data222for various tasks. In some embodiments, the vision task228can be integrated with the vision application230to exchange data and control commands. In some embodiments, the system200can connect to one or more entities250that can implement different image related processes, that can receive image data222and bounding box data226, and/or that can display the interface with virtual tool224, for example. The system200can connect to data sources to receive image data222, for example.

The I/O unit102can enable the system100to interconnect with one or more input devices, such as a keyboard, mouse, camera, touch screen and a microphone, and/or with one or more output devices such as a display screen and a speaker. An input device can be used to control the virtual tool224at the interface and define bounding box data226relative to image data222displayed at the interface. The input device can be used to generate touch input data and movement data, for example.

The interface controller220can trigger the display of a user interface on a display device. The user interface can display an image (or a portion thereof) from image data222. The user interface can enable selection of the image (or a portion thereof) from image data222. The interface controller220can activate the virtual tool224at the interface to define a BB. The virtual tool224can be controlled by commands received from interaction between the input device and the interface. The interface controller220can trigger the display of an indicator for the virtual tool relative to the image data. The virtual tool224can be displayed as a graphical object such as a pointer, marker, and the like. The interface controller220can trigger the display of a graphical representation of a source image and a graphical representation of the indicator for the virtual tool relative to the image data.

The interface controller220can receive, from the interface, a first input data point at a first location relative to the image (or a portion thereof). The capture of the first input data point data can be triggered by actuation of the input device. This can be a click, selection, or a touch of the input device, for example. The first input data point can define a corner location for the BB. The input device can be integrated with a touch display, for example, and the first input data point can be referred to as a first touch input.

The interface controller220can receive, from the interface, movement input in a direction relative to the first location. The movement input can be defined by movement commands from the input device during the actuation of the input device and a release of the actuation of the input device. This can be a drag or swipe from the first corner location, for example. The movement input can define an edge angle from (and relative to) the corner location. The movement input can be in the direction of the adjacent corner for the BB, for example.

The interface controller220can receive, from the interface, a second input data point at a second location relative to the image. The capture of the second input data point can be triggered by actuation of the input device. This can be a click, selection, or a touch of the input device, for example. The second input data point can define another corner location for the BB. The second input data point can define the location of the opposite corner to the first corner of the BB, for example.

The interface controller220can trigger display, at the interface, of a graphical object representing the BB as an overlay of the image data. The BB corners are defined by the first location and the second location. In particular, the first location indicates a corner of the BB and the second location can indicate the opposite corner of the BB. The BB has edges connecting adjacent corners. The edges can be defined by the direction of the movement input, and in particular, the angle of the edges relative to adjacent corners can be based on the movement input.

The interface controller220can receive the input data from the interface and transform the input data into a BB data format. The interface controller220can compile the input into code representing the BB data format. The transformed input data can define a BB record. The BB record can be linked with the image. For example, the BB record can include metadata that includes an image identifier, for example. The interface can store the BB record as bounding box data226. Different metadata can be stored in association with the BB. For example, the metadata can be the actual text written in the box if it is assumed to be text, the class of object e.g. car, truck, human, information about the BB itself e.g. partial, occluded, and so on.

The interface can receive a command from the input device indicating actuation of virtual tool224(e.g. first input data point) at a first location and movement of the virtual tool224from the first location to another location (movement input) while the virtual tool224is continuously actuated (e.g. click, hold click while dragging input device to a new location). For example, the input data can be defined as a first click of the input device and drag of the virtual tool224from P1,startto P1,end. The interface can receive a command indicating that the virtual tool224is released (e.g. no longer actuated). The interface can receive another command indicating another actuation of the virtual tool224(e.g. second input data point) at a second location. For example, the input data can be defined as a second click at P2. The interface can provide the input data to the interface controller220to compute the BB.

The interface controller220can use the input data to compute the BB angle θ (e.g. angle of the edge between adjacent corners). The angle of first drag can be computed as follows:
A:=P1,end−P1,start
∠A:=atan 2(Ay,Ax)

The interface controller220can wrap the angle between −45° and 45°
θ:=((∠A+45°)mod 90°)−45°

The interface controller220can compute the BB center coordinates C:

The interface controller220can compute the BB extent E (half-width, half-height). For example, the interface controller220can compute the BB extent E by computing the diagonal from center to P2, or alternatively half the full diagonal.

If θ is different from 0, the interface controller220can rotate E in the opposite angle −θ. This can be a change of basis towards a basis defined by the first input data point (click) and movement input (drag). This can be performed by multiplying a rotation matrix.

The interface controller220can take the absolute value of each coordinate

Considering an example with the origin (e.g. first input data point) at the bottom-left corner, the following indicates computations for the OOBB example depicted inFIG. 3.

The interface controller220can receive the following input data from actuation and movement of the virtual tool224. The first input data point and the movement data can be:

The second input data point can be:

The interface controller220can compute the BB angle θ by computing the angle of the first drag (movement of the virtual tool224within the interface)

As shown, wrapping the angle between −45° and 45° does not have an effect
θ:=((36.9°+45°) mod 90°)−45°=(81.9° mod 90°)−45°=81.9°−45°=36.9°

The interface controller220can compute BB center coordinates C:

The interface controller220can compute BB extent E (half-width, half-height). For example, the interface controller220can compute diagonal from center to P2

The interface controller220can rotate E in the opposite angle −θ.

The interface controller220can take the absolute value of each coordinate.

The virtual tool224may draw or define the same OOBB by specifying the same two corners but the virtual tool224can be dragged to define the other edge. That is, different orders of input data points can be used to define the same OOBB.FIG. 5illustrates an example BB102around text100that can be defined by a first input data point502(for a first corner location), a movement input504of the virtual tool224(for the edge angle), and a second input data point506(for the second corner location). The difference in computation can be limited to up until the angle is wrapped between −45° and 45°.

From the control commands generated at interface using the virtual tool224, the interface controller220can receive the following example input data. The interface controller220can receive the first input data point (click) and movement input (drag) of the virtual tool224from

The interface controller220can receive the second input data point (click) at P2=

The interface controller220can compute the BB angle θ. The interface controller220can compute the angle of first movement input (drag)

Wrapping the angle between −45° and 45° can compute the same angle as the example described in relation toFIG. 3.
θ:=((−53.1°+45°) mod 90°)−45°=(−8.1° mod 90°)−45°=81.9°−45°=36.9°

The virtual tool224may draw or define the same OOBB by starting by a completely different corner (e.g. the first input data point can be at a different corner location). The difference in computation carries up until the absolute value is finally performed on the extent.FIG. 6illustrates an example BB102around text100that can be defined by a first input data point602(for a first corner location), a movement input604of the virtual tool224(for the edge angle), and a second input data point606(for the second corner location).

From the control commands generated at interface using the virtual tool224, the interface controller220can receive the following example input data. The interface controller220can receive the first input data point (click) and movement input (drag) of the virtual tool224from

The interface controller220can receive the second input data point (click) at

The interface controller220can compute the BB angle θ. The interface controller220can compute the angle of first movement input (drag)

Wrapping the angle between −45° and 45° can return the same angle
θ:=((−126.9°+45°) mod 90°)−45°=(−98.1° mod 90°)−45°=81.9°−45°=36.9°

The interface controller220can compute the BB center coordinates C:

The interface controller220can compute the BB extent E (half-width, half-height). The interface controller220can compute the diagonal from center to P2which indicates coordinates with signs that are different, but consistent

The interface controller220can rotating E in the opposite angle −θ.

The interface controller220can compute the absolute value of each coordinate which returns the same extents.

Accordingly, the virtual tool224provides a flexible tool to define BB based on a set of input data that can define different corners and edge angles for the BB. The input data can be efficiently provided by the user to the system200.

The processor204can save a bounding box record (as part of the bounding box data226) for the bounding box with metadata with an identifier for the image data that was displayed at interface with the bounding box was defined using the virtual tool224.

The processor204can compute the bounding box in a bounding box format using the first input data point, the movement input, and the second input data point. For example, the format can define a first corner at the first location, an opposite corner at the second location, and an adjacent corner connected to the first corner by an edge at an angle defined by the movement input. As another example, the bounding box format is an angle-center-extent format defined by an angle of an edge between a corner and an adjacent corner, centre coordinates of the bounding box, and an extent of the bounding box.

The movement input can be from the first location towards an adjacent corner. Accordingly, the direction of the movement input being towards an adjacent corner of the bounding box. The second location can indicate the opposite corner.

The processor204can dynamically update the interface to display a line between the first location and a current location of the virtual tool when defining the movement input. The processor204can dynamically update the interface to display a suggested bounding box defined by the first location, the movement input and a current location of the virtual tool when defining the second input data point and prior to the second actuation of the input device.

The processor204can extract a region of interest from the image data defined by the bounding box, and save the extracted region of interest in data storage110as part of the image data222. The processor204can transmit the extracted region to vision task228, vision application230, or entity250, for example.

The processor204can be, for example, any type of general-purpose microprocessor or microcontroller, a digital signal processing (DSP) processor, an integrated circuit, a field programmable gate array (FPGA), a reconfigurable processor, or any combination thereof.

The system200can be operable to register and authenticate users (using a login, unique identifier, and password for example) prior to providing access to applications, a local network, network resources, other networks and network security devices. The system200can connect to different machines, entities250, and/or data sources260(linked to databases270).

The data storage210may be configured to store information associated with or created by the system200, such as for example image data222and bounding box data226. The data storage210may be a distributed storage system, for example. The data storage210can implement databases, for example. Storage210and/or persistent storage114may be provided using various types of storage technologies, such as solid state drives, hard disk drives, flash memory, and may be stored in various formats, such as relational databases, non-relational databases, flat files, spreadsheets, extended markup files, and so on.

FIG. 7is a diagram of another example OBB that can be defined using a three-step workflow. Embodiments described herein provide an improved workflow and this three-step workflow is described for contrast. The three-step workflow involves two clicks to define two adjacent corners (e.g. corners702,704) and a third click defines the perpendicular dimension706of the BB. This three-step workflow requires additional actuations of the input device than the improved workflow of embodiments described herein. Further, the three-step workflow does not easily allow for an AABB, necessitating a different tool or workflow. In contrast, embodiments described herein can be used to define different types of BB such as an OBB and an AABB, avoiding the need for different tools.

The embodiments described herein are implemented by physical computer hardware, including computing devices, servers, receivers, transmitters, processors, memory, displays, and networks. The embodiments described herein provide useful physical machines and particularly configured computer hardware arrangements.

Although the embodiments have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein.

Moreover, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification.