Digital image processing system for object location and facing

Despite the impressive advances made in recent decades, past digital image processing system were faced with significant technical challenges to solving important technical problems. The digital image processing system described below helps to solve these technical challenges with regard to spatial location and orientation of arbitrary objects in real-world environments. The digital image processing system performs image segmentation to accurately identify objects in an image, then locates the objects and determines their orientations.

TECHNICAL FIELD

This disclosure relates to image processing. More particularly, this disclosure relates to processing image data to identify objects within the images as well as the location and facing of those objects.

BACKGROUND

Digital image processing has a tremendous range of application and for some important problems is the only practical approach at a solution. Despite the advances made in the last several decades with image processing, significant technical challenges remain in many subject matter areas and for many applications. Improvements in digital image processing systems will help solve these technical challenges, e.g., with regard to identification, spatial location, and orientation of arbitrary objects captured in images of real-world environments.

DETAILED DESCRIPTION

Important technical problems pose a challenge to digital image processing systems. As one example, the technical problems include accurately identifying discrete objects in a scene which may be very simple or very complex, while at the same time being subject to a wide range of image quality considerations, such as obscuring shadows. Additional technical problems include determining the spatial location of the objects, e.g., their latitude and longitude, as well as their orientation, e.g., angles of rotation. The digital image processing system described below solves these and other technical problems.

FIG. 1shows an arbitrary environment100and a digital image processing system102(“system102”) operating to determine spatial location and orientation of objects in the environment100. This example is used for discussion purposes below; the system102may analyze any environment including any objects. In this example, the environment100includes a parking lot104and objects including cars106, stores108, billboards110and112, and utility poles114. A road116runs near the parking lot104.

Images of the parking have been taken along the road from specific viewpoints. InFIG. 1, four viewpoints118,120,122, and124are shown that capture certain portions of the parking lot104and certain objects within the parking lot104. The viewpoints118-124are located along the road116(and may be considered streetview viewpoints of the parking lot), but the viewpoints may be at vantage point whether on or along the street or elsewhere.

An image data provider126receives the image data captured at the viewpoints118-124. The image data may be tagged with geolocation, timestamp, and other metadata. The image data provider126builds and maintains the image data database128, e.g., by updating or adding the image data from the viewpoints118-124to existing image data in the image data database128.

Similarly, a road data provider130builds and maintains a road graph database132. The road graph database132stores graph representations of road networks in any given geographical regions for which the road data provider130supplies data. In this example, the road data provider130maintains a road graph136of the road network in the environment100.

The system102communicates over the networks138with the image data provider126, e.g., to receive image data134, and with the road data provider130, e.g., to receive road graphs such as the road graph136. The system102includes communication interfaces140, image processing circuitry142, local databases144, and models146. The local databases144may store image data, road graphs, and object data, for instance. The models146may include, as examples, scene segmentation models that predict object labels, linear smoothing models to smooth mask boundaries, object identification models that identify obstructing foreground objects, object splitting models that separate overlapping objects, and mask accuracy classifiers to validate mask accuracy.

The system102applies the models146to determine the spatial location of selected objects and their orientations in the environment100. The system102may then transmit object reports148conveying the spatial locations and orientations to any desired external object systems150. The object systems150may represent, for instance, systems or organizations responsible for the placement, monitoring, or management of any of the objects in the environment100. Accordingly, the object system150may verify the proper placement and orientation of the objects for which they are responsible, or may initiate corrective actions based on the object reports148, e.g., actions to correct an incorrect placement or orientation. In addition, the object systems150may provide object data152to the system102as described in more detail below.

FIG. 2shows an example of the logic200that the system102may execute. The system102receives object data202, e.g., from the object systems150(204). The object data202specifies expected properties of specific objects in the environment100. For instance, the object data202may specify the correct or expected object orientation (e.g., angle(s) of rotation); expected object location (e.g., latitude and longitude); expected object elevation; object width, depth, and height; or other object properties for any object in the environment100. In the example inFIG. 2, the system102receives object data for the billboard110that specifies an expected location for the billboard110and an expected orientation for the billboard110in terms of rotation around a vertical axis.

The system102also receives image data206from multiple distinct viewpoints in the environment100(208). The image data206captures images of the environment100in which the object under analysis is expected to exist. Among other characteristics, the image data206may be characterized by longitude, latitude, heading, and field of view (which indicates focal length) looking at those objects that the system102will analyze. Expressed another way, the system102receives image data capturing images of one or more of the objects in the environment100for which the system102received the object data202. The image data provider126may transmit the image data to the system102at the request of the system102, for instance. Alternatively, the system102may obtain the image data from its own local database144, or from another image source.

The system102segments the images to find individual objects within the images (210). With regard to the example ofFIG. 1, the system102may determine that (at least) four distinct objects exist in the images obtained from viewpoint122and viewpoint124. Having determined that there are four objects, the system102may then identify those objects (212). In this example, the system102determines that the four objects are (or are likely to be) the billboard110, billboard112, utility pole114, and car106.

For any of the objects, the system102may determine a measured object location from the image data, e.g., the latitude and longitude of the object (214). In addition, the system102may determine measured object orientation(s) from the image data, e.g., one or more angles of rotation around predefined axes of that object at its location (216). The system102inserts the measured object location and measured object orientation data into object reports148and transmits the object reports148to the object systems150responsible for the objects (218).

The analyses performed by the system102are described in more detail below inFIGS. 3 and 4.FIG. 3shows an example of logic300that the system102may execute, including determining sample points for image data and segmenting images. InFIG. 3, the system102has received the object data (204) which includes the expected object location, orientation, and other properties. The system102requests and receives a road graph302for the environment100in which the object exists (304), e.g., from the road data provider130. Alternatively, the system102may obtain the road graph from its own local database144, or from other sources. Given the expected location of the object specified in the object data202, the system102determines a reference point306from which the system102will determine view sample points. For instance, the system102may choose the point on the nearest road that minimizes distance to the object (308) as the reference point306.

From the reference point306, the system102moves along the road in opposite directions for a predetermined distance, e.g., 30 meters, a predetermined number of times, e.g., one 30 meter offset in each direction. These offsets give multiple sample points310and312(e.g., specified by latitude and longitude) from which to obtain image data of the environment100. Based on the sample points, the system102requests and receives image data from the image data provider126. That is, the system102requests images with a camera heading oriented toward the subject (316) at the sample points310and312or within a system configurable radius of the sample points310and312. For the example environment100, the sample points310and312may result in image data received from the viewpoints122and124for the billboard object110, for instance.

As noted above with regard toFIG. 2, given the image data the system102segments the images to find objects (210). In more detail, the system102provides the images as input to a trained scene segmentation model318(320). In one implementation, the scene segmentation model318predicts object labels for each pixel in the images (322). The labels form masks that the system102may save as single color channel images (e.g., portable network graphic (PNG) images) (324). In these images, the pixel intensity may match to any predetermined image class predefined or preconfigured within the system102.

The system102also applies a linear model326to obtain masks with smoothed boundaries (328). In one implementation, the linear model326detects linear boundaries from mask outlines and detects the front or face of an object based on color contrast. The scene segmentation model318and linear model326help to significantly eliminate shadows and other image artifacts in the images. For example, the models will eliminate shadows cast by poles, catwalks, and other supporting structure, as well as eliminate shadows cast by a second object that is back-to-back with a first object.

In one implementation, the scene segmentation model318is a deep neural network such as a wide residual neural network. This network may be trained, e.g., on 10,000 or more street scenery images with 200 or more object classes. The predetermined image classes include, as examples, different types of ground surfaces such as roads and sidewalks; naturally occurring objects such as the sky, plants, grass, and water; man-made objects such as poles, buildings, benches, street signs, business signs, and billboards; types of road lane markings; vehicles such as cards, busses, bicyclists, and motorcycles; living creatures such as humans, birds, dogs, and cats; and other classes.

In one implementation, the system102assumes that object boundaries are linear or are combinations of linear boundaries. The linear model326may then use the linear (and narrow) distribution of boundary pixels and sharp color contrast to filter out fluctuating pixel noise and error coming from the masks output by the scene segmentation model318. With the knowledge of possible geometric shapes from the 3D object as projected into 2D imagery (e.g., a trapezoid), the system102further filters out the shadows.

Building on the prior examples inFIGS. 2 and 3,FIG. 4shows an example of logic400that the system102may execute, including identifying objects and determining object location and orientation. The system102applies a foreground object identification model402to the masks (404). The foreground object identification model402identifies objects that are in the foreground, and the system102removes the foreground objects from the masks (406). The system102also applies a shared structure model408. The shared structure model408splits composite objects into separate objects (410). Examples of composite objects include objects sharing a structure or overlapping in two dimensions.

Next, the system102provides each proposed localized object to a validation classifier412which validates the accuracy of the masks (414). The validation classifier412generates a series of candidate mask crops for each mask (416), e.g., those output from (328) explained above. In one implementation, the validation classifier412is a deep neural network, e.g. a deep residual convolutional network trained over many types of object labels, e.g., 15,000 or more object labels. The mask crops capture identified objects that have been segmented out of the image data206. That is, the single channel color mask contains the cropped out pixels of the identified object from the original image.

In one implementation, the validation classifier412is a deep neural network such as a residual convolutional neural network. The validation classifier412may be trained on 10,000 or more images to classify whether that class of object exists in an image. The shared structure model408takes the mask crops output from the scene segmentation model318and transforms the cropped imagery to a single channel gray scale image. Based on pixel luminance contrast, the shared structure model408may separate two or more objects on the same structure into a separate mask for each object. The foreground object identification model402determines the probability of an object being in the foreground of another object. In one implementation, the foreground object identification model402performs its analysis using the pixel 2D continuity of PNG results from the scene segmentation model318.

The system102estimates the focal length of the camera based on the field of view of the images (418). In some implementations and for some objects, the system102may assume that the left side edge and right side edge of the object should stand vertically on the ground, and that the surface of the road is parallel to the ground under the object. Further, the system102may also assume that there should be negligible rotations of the object side edges in the vertical direction. Under these assumptions, the system102may estimate the length per-pixel at the location of the object (420) given the physical height of the object from the object data202, e.g., by counting the number of pixels along the height of the object.

In addition, the system102may estimate the deviation of the object from the camera heading (422) toward the object. In that respect, note that the system102will have an estimate of the distance from the known sample points to the object location. The system102may therefore count the pixel distance from the center of the object to the center of the image and converts the distance into a physical length in order to estimate the deviation of the object from the camera heading.

With regard to the location of the object, the system102may determine bearings to the object from each sample point (424). Continuing the example above, the system102may, for instance, determine a bearing426to the billboard110for viewpoint122and a bearing428to the billboard110for viewpoint124. The system102determines the intersection of the bearings from the different viewpoints as the location of the object, e.g., in latitude and longitude (430).

With regard to object orientation, the system102estimates the angular rotation of the object around any chosen axis, e.g., a yaw angle ϕ around a vertical axis. The system102may perform the estimation by determining object width, e.g., as the rotated pixel length450from the left side edge of the object to the right side edge of the object in the image (432), and determining the unrotated pixel length452(or object width) if the object were facing without rotation toward the camera (434). The difference in pixel length is a function of the angle of rotation and known object shape, from which the system102determines the relative angle of rotation (436).

The system102also determines the direction of rotation with respect to the camera heading (438). In one implementation, the system102makes this determination based on which edge of the object is higher than the other, e.g., if the left edge is higher than the right edge then the rotation is counterclockwise. Adding or subtracting (depending on direction of rotation) the determined direction of rotation to or from the camera heading gives the overall angular rotation ϕ of the object along the chosen axis (440).

FIG. 5shows an example implementation500of the system102. The implementation500includes communication interfaces502, system circuitry504, input/output (I/O) interfaces506, and display circuitry508. The system circuitry504may include any combination of hardware, software, firmware, or other circuitry. The system circuitry504may be implemented, for example, with one or more systems on a chip (SoC), application specific integrated circuits (ASIC), microprocessors, microcontrollers, discrete analog and digital circuits, and other circuitry. The system circuitry504is part of the implementation of any desired functionality in the system102. Accordingly, the system circuitry504may implement the logic200,300, and400described above with regard toFIGS. 2-4, as examples. The system102may store and retrieve data from data memories516. For instance, the data memories516may store the local databases144and the models146.

The display circuitry508and the I/O interfaces506may include a graphical user interface, touch sensitive display, voice or facial recognition inputs, buttons, switches, speakers and other user interface elements. Additional examples of the I/O interfaces506include Industrial Ethernet, Controller Area Network (CAN) bus interfaces, Universal Serial Bus (USB), Serial Advanced Technology Attachment (SATA), and Peripheral Component Interconnect express (PCIe) interfaces and connectors, memory card slots, and other types of inputs. The I/O interfaces506may further include Universal Serial Bus (USB) interfaces, audio outputs, magnetic or optical media interfaces (e.g., a CDROM or DVD drive), network (e.g., Ethernet or cable (e.g., DOCSIS) interfaces), or other types of serial, parallel, or network data interfaces.

The communication interfaces502may include transceivers for wired or wireless communication. The transceivers may include modulation/demodulation circuitry, digital to analog converters (DACs), shaping tables, analog to digital converters (ADCs), filters, waveform shapers, filters, pre-amplifiers, power amplifiers and/or other circuitry for transmitting and receiving through a physical (e.g., wireline) medium such as coaxial cable, Ethernet cable, or a telephone line, or through one or more antennas. Accordingly, Radio Frequency (RF) transmit (Tx) and receive (Rx) circuitry510handles transmission and reception of signals through one or more antennas512, e.g., to support Bluetooth (BT), Wireless LAN (WLAN), Near Field Communications (NFC), and 2G, 3G, and 4G/Long Term Evolution (LTE) communications.

Similarly, the non-wireless transceivers514may include electrical and optical networking transceivers. Examples of electrical networking transceivers include Profinet, Ethercat, OPC-UA, TSN, HART, and WirelessHART transceivers, although the transceivers may take other forms, such as coaxial cable network transceivers, e.g., a DOCSIS compliant transceiver, Ethernet, and Asynchronous Transfer Mode (ATM) transceivers. Examples of optical networking transceivers include Synchronous Optical Networking (SONET) and Synchronous Digital Hierarchy (SDH) transceivers, Passive Optical Network (PON) and Ethernet Passive Optical Network (EPON) transceivers, and EPON Protocol over Coax (EPoC) transceivers.

Note that the system circuitry504may include one or more controllers522, e.g., microprocessors, microcontrollers, FGPAs, GPUs, and memories524. The memories524store, for example, an operating system526and control instructions528that the controller522executes to carry out desired functionality for the system102. Accordingly, the control instructions528may implement the logic200,300, and400described above and with regard toFIGS. 2-4for object location determination, object orientation determination, and object report preparation and transmission. The control parameters530provide and specify configuration and operating options for the control instructions528.

The implementations may be distributed. For instance, the circuitry may include multiple distinct system components, such as multiple processors and memories, and may span multiple distributed processing systems. Parameters, databases, and other data structures may be separately stored and managed, may be incorporated into a single memory or database, may be logically and physically organized in many different ways, and may be implemented in many different ways. Example implementations include linked lists, program variables, hash tables, arrays, records (e.g., database records), objects, and implicit storage mechanisms. Instructions may form parts (e.g., subroutines or other code sections) of a single program, may form multiple separate programs, may be distributed across multiple memories and processors, and may be implemented in many different ways. Example implementations include stand-alone programs, and as part of a library, such as a shared library like a Dynamic Link Library (DLL). The library, for example, may contain shared data and one or more shared programs that include instructions that perform any of the processing described above or illustrated in the drawings, when executed by the circuitry.