Iterative layout mapping via a stationary camera

Disclosed herein are apparatuses and methods for iteratively mapping a layout of an environment. The implementations include receiving a visual stream from a camera installed in the environment, wherein the visual stream depicts a view of the environment, and wherein positional parameters of the camera and dimensions of the environment are set to arbitrary values. The implementations include monitoring a plurality of persons in the visual stream. For each person in the plurality of persons, the implementations further includes identifying a respective path that the person moves along in the view, updating the dimensions of the environment captured in the view, based on an estimated height of the person and movement speed along the respective path, and updating the positional parameters of the camera based on the updated dimensions of the environment. The implementations further includes mapping a layout of the environment captured in the view of the camera.

TECHNICAL FIELD

The described aspects relate to mapping systems.

BACKGROUND

Aspects of the present disclosure relate generally to mapping systems, and more particularly, to iteratively mapping the layout of an environment using a stationary camera.

Simultaneous localization and mapping (SLAM) algorithms are often used to determine the layouts of environments right down to their exact measurements. In SLAM, a camera/depth sensor is guided along a path in an environment that ideally has stationary objects. Using the information collected along the path and knowing how fast the camera/depth sensor was moving or how far the camera/depth sensor moved, a detailed layout of the environment can be generated.

There are situations, however, where a layout of an environment may be desired while the camera is kept stationary. For example, a security camera may be installed in an arbitrary position in a shopping mall and landmarks such as walkways, doors, etc., may need to be identified. Information such as the physical parameters (e.g., installation height, tilt angle, etc.) of the camera may be unknown. Here, SLAM algorithms will be ineffective for mapping because the security camera does not move. Furthermore, simple image classification to identify walkways and doors will be ineffective because the measurements of the walkways and doors would not be determined.

Accordingly, there exists a need for improvements in mapping systems.

SUMMARY

An example implementation includes a method for iteratively mapping a layout of an environment, comprising receiving a visual stream from a camera installed in the environment, wherein the visual stream depicts a view of the environment, and wherein positional parameters of the camera and dimensions of the environment are set to arbitrary values. The method further monitoring a plurality of persons in the visual stream. For each person in the plurality of persons, the method further includes identifying a respective path that the person moves along in the view, updating the dimensions of the environment captured in the view, based on an estimated height of the person and movement speed along the respective path, and updating the positional parameters of the camera based on the updated dimensions of the environment. The method further includes mapping a layout of the environment captured in the view of the camera.

Another example implementation includes an apparatus for iteratively mapping a layout of an environment, comprising a memory and a processor configured to communicate with the memory. The processor is configured to receive a visual stream from a camera installed in the environment, wherein the visual stream depicts a view of the environment, and wherein positional parameters of the camera and dimensions of the environment are set to arbitrary values. The processor is configured to monitor a plurality of persons in the visual stream. For each person in the plurality of persons, the processor is configured to identify a respective path that the person moves along in the view, update the dimensions of the environment captured in the view, based on an estimated height of the person and movement speed along the respective path, and update the positional parameters of the camera based on the updated dimensions of the environment. The processor is configured to map a layout of the environment captured in the view of the camera.

Another example implementation includes an apparatus for iteratively mapping a layout of an environment, comprising means for receiving a visual stream from a camera installed in the environment, wherein the visual stream depicts a view of the environment, and wherein positional parameters of the camera and dimensions of the environment are set to arbitrary values. The apparatus further includes means for monitoring a plurality of persons in the visual stream. Additionally, the apparatus further includes means for, for each person in the plurality of persons, (1) identifying a respective path that the person moves along in the view, (2) updating the dimensions of the environment captured in the view, based on an estimated height of the person and movement speed along the respective path, and (3) updating the positional parameters of the camera based on the updated dimensions of the environment. Additionally, the apparatus further includes means for mapping a layout of the environment captured in the view of the camera.

Another example implementation includes a computer-readable medium for iteratively mapping a layout of an environment, executable by a processor to receive a visual stream from a camera installed in the environment, wherein the visual stream depicts a view of the environment, and wherein positional parameters of the camera and dimensions of the environment are set to arbitrary values. The instructions are further executable to monitor a plurality of persons in the visual stream. For each person in the plurality of persons, the instructions are further executable to identify a respective path that the person moves along in the view, update the dimensions of the environment captured in the view, based on an estimated height of the person and movement speed along the respective path, and update the positional parameters of the camera based on the updated dimensions of the environment. Additionally, the instructions are further executable to map a layout of the environment captured in the view of the camera.

DETAILED DESCRIPTION

The present disclosure includes apparatuses and methods that map the layout of environment via a camera that can be installed in any arbitrary position in the environment. Unlike SLAM, where the camera is moved and all other objects in the environment remain stationary, in the present disclosure the installed camera remains stationary and relies on the movement of people to estimate distance and the dimensions of the environment. This allows for a greater use of cameras that are installed in one position such as security cameras in offices, supermarkets, etc.

FIG. 1is a diagram of a scenario for mapping a layout of environment100, in accordance with exemplary aspects of the present disclosure. Suppose that environment100is a grocery store and camera102is installed near the ceiling of environment100. The visual stream captured by camera102, which may be a video or a series of periodic images marked by timestamps, may be transmitted to computing device400. Computing device400may be a computer, a laptop, a smartphone, a server, or any device capable of receiving the visual stream from camera102and processing it using a layout mapping component415(discussed inFIGS. 4-6). Computing device400may be located in environment100or away from/outside of environment100. Furthermore, camera102may be connected to computing device400wirelessly (e.g., via Bluetooth, Wi-Fi, etc.) or through a wired connection (e.g., USB).

Environment100may include door112and landmarks110(e.g., shelves, fridges, racks, etc.). A plurality of persons such as person104and person106may walk through environment100. Person104and106may be employees, customers, managers, security officers, etc. When walking, for example, person104may take path108, which represents the route person104takes as captured by camera102. For simplicity, path108is extended to the right edge of environment100(even though person104has only walked mid-way. This is to show the complete route person104takes while being captured by camera102.

FIG. 2is a diagram of view200of camera102inFIG. 1, in accordance with exemplary aspects of the present disclosure. As can be seen, the field of vision (FoV) of camera102is bound in a rectangular shape. Accordingly, camera102can only capture person104and106, a portion of door112, a portion of path108, and a portion of landmarks110. Furthermore, from the perspective of camera102, person104is walking towards camera102. Because camera102is placed in an arbitrary position in environment100, the positional parameters of camera102are unknown (unless specified by a user that installed the camera or has the positional parameters). The positional parameters of camera102comprise at least one of: camera installation height, tilt angle, FoV, or focal length. Because all objects depicted in view200are nearby or far away relative to camera102, determining the positional parameters is necessary to generate a layout of environment100. By determining the positional parameters, the dimensions of environment100can be determined relative to camera102. In the present disclosure, these values are updated iteratively to generate a more accurate layout of environment100(as viewed in view200). As more persons enter and exit the view of environment100, computing device400is better able to determine the depth, width, and height of environment100based on how quickly the persons walk and their approximate heights.

FIG. 3is a diagram of layout300of environment100inFIG. 1, in accordance with exemplary aspects of the present disclosure.FIG. 5will later describe mapping layout300of environment100. However, as shown inFIG. 3, pathway302is visually labelled and so is ingress/egress point304. Pathway302represents an aggregate of all routes taken by the plurality of persons in environment100. Ingress/egress point304represents a portion of view200(used interchangeably with frame200) in which persons appear and disappear from the FoV. Pathway302and point304may be labeled with distances and measurements (e.g., the width of pathway302is 5 feet).

In some aspects, layout300may be used to identify pathways in environment100that are most often used. This allows someone that monitors environment100(e.g., an owner of a store) to notice hot spots where several people gather. If those people are customers, the owner may consider rearranging the stock of items to place more popular items near the hot spots for easier access. In some cases, the owner may need to know whether people are following rules in environment100. For example, if social distancing rules are in place and people need to be 6 feet apart, the owner may determine whether the aisles need to be rearranged because people are too close and violate the rules too often. Thus, layout mapping component415of the present disclosure may offer several real-world benefits beyond providing a technically-improved apparatus and method of mapping.

FIG. 4is a block diagram of computing device400executing an layout mapping component415, in accordance with exemplary aspects of the present disclosure.FIG. 5is a flowchart illustrating method500of iteratively mapping a layout of an environment, in accordance with exemplary aspects of the present disclosure. Referring toFIG. 4andFIG. 5, in operation, computing device400may perform method500of iteratively mapping a layout of an environment via execution of layout mapping component415by processor405and/or memory410.

At block502, the method500includes receiving a visual stream from a camera installed in the environment, wherein the visual stream depicts a view of the environment, and wherein positional parameters of the camera and dimensions of the environment are set to arbitrary values. For example, in an aspect, computer device400, processor405, memory410, layout mapping component415, and/or receiving component420may be configured to or may comprise means for receiving a visual stream from camera102installed in environment100, wherein the visual stream depicts view200of environment100, and wherein positional parameters of the camera and dimensions of environment100are set to arbitrary values.

As mentioned before, suppose that the position parameters of camera102are camera installation height, tilt angle, field of vision (FoV), and/or focal length. Layout mapping component415may set each parameter to an initial value (e.g., installation height is 5 feet, tilt angle is 20 degrees downward relative to the wall of environment100where camera102is installed, field of vision is 40 feet, etc.). Because layout mapping component415does not actually know the real values of these parameters, they are updated iteratively during method500. In particular, layout mapping component415receives the visual stream from camera102and identifies people in the received frames of the stream.

At block504, the method500includes monitoring a plurality of persons (N) in the visual stream. For example, in an aspect, computer device400, processor405, memory410, layout mapping component415, and/or monitoring component425may be configured to or may comprise means for monitoring a plurality of persons (e.g., person104, person106) in the visual stream.

Suppose that view200represents a frame of the visual stream received by layout mapping component415. Monitoring component425may utilize computer vision techniques such as person recognition to identify person104and person106in the frame. In response to detecting person104and person106, monitoring component425may track the movements of each person. For example, over a plurality of frames (e.g., 300 frames provided over 10 seconds), person104may move along path108and person106may enter through door112and stand still.

At block506, the method500includes identifying person (i) of N. For example, in an aspect, computer device400, processor405, memory410, layout mapping component415, and/or monitoring component425may be configured to or may comprise means for identifying person104in the visual stream.

For example, layout mapping component415may analyze each person identified one at a time. The value of N represents the total number of identified persons and (i) represents the number of the person. Initially the value of (i) may be 1, indicating that the first person is being identified (e.g., person104). In this example, the value of N is 2 for simplicity. However, N may change as more individuals are captured by camera102over time. In addition, layout mapping component415may set the initial estimated height and movement speed of person104to predetermined values. The predetermined values may be dependent on the region in world that the camera102is located in. For example, the average height for a man in America is 5 feet 9 inches and the average walking speed is 3 miles per hour.

At block508, the method500includes identifying a respective path that the person moves along in the view. For example, in an aspect, computer device400, processor405, memory410, layout mapping component415, and/or identifying component430may be configured to or may comprise means for identifying path108that person104moves along in view200.

In some aspects, identifying component430may treat each person as a point and monitor how the point travels across the plurality of frames that the point appears in. For example, a point on person104may be on the head of person104. Identifying component430may, in some aspects, create a three-dimensional x-y-z representation of environment100that is relative to the walls and floor captured in view200. For example, the wall near person106includes door112. The height-wise line of the door may be characterized as parallel to the z-axis. The width-wise line of the door may be characterized as parallel to the x-axis. The y-axis may be determined as the axis perpendicular to both the x-axis and the z-axis. Path108can therefore be conveyed as a combination of vectors. The first vector is parallel to the x-axis and the second vector is parallel to the y-axis.

At block510, the method500includes updating the dimensions of the environment captured in the view, based on an estimated height of the person and movement speed along the respective path. For example, in an aspect, computer device400, processor405, memory410, layout mapping component415, and/or updating component435may be configured to or may comprise means for updating the dimensions of environment100(e.g., lengths along the x-axis, y-axis, and z-axis) captured in view200, based on an estimated height of person104and movement speed along path108.

For example, the first portion of path108is parallel to the x-axis. If person104walks along the first portion for 2 seconds and is estimated to walk 4 feet per second, the length of the vector representing the first portion of the path is determined by layout mapping component415as 8 feet. Updating component435can extrapolate this information by determining that the first portion represents a third of the pixels along the x-axis. Therefore, the environment100is at least 24 feet along the x-axis.

As person104walks along the second portion of path108(e.g., the line perpendicular to the y-axis, person104may walk for 4 seconds. Accordingly, layout mapping component415may determine that the second portion is 16 feet in length. Extrapolating on this data, updating component435may determine that the second portion represents 80% of the pixels along the y-axis, which would mean that environment100is at least 20 feet along the y-axis.

Furthermore, because the height of person104is estimated to be 5 feet 9 inches (the predetermined value) along the z-axis, updating component435may determine the amount of pixels along the z-axis that person104measures out to be and may extrapolate the height of environment100. For example, updating component435may extend a line from the feet of person104to the top of a frame (along the z-axis) and determine the number of pixels in the line that are of person104. Suppose that only 50% of the pixels in the line are of person104. This would mean that a remaining 5 feet 9 inches above person104represents environment100. Accordingly, updating component435may determine that the height of environment100is at least 11.5 feet.

In some aspects, knowing the physical length (i.e., real life length) a group of pixels represents, updating component435may determine the measurements of various objects in view200. For example, updating component435may determine the dimensions of landmarks110.

At block512, the method500includes updating the positional parameters of the camera based on the updated dimensions of the environment. For example, in an aspect, computer device400, processor405, memory410, layout mapping component415, and/or updating component440may be configured to or may comprise means for updating the positional parameters (e.g., tile angle, installation height, FoV, focal length) of camera102based on the updated dimensions of environment100.

For example, updating component440may utilize a machine learning algorithm that is trained to output a tilt angle, a camera installation height, a FoV, a focal length, or any combination thereof. The machine learning algorithm may be trained on a dataset that includes a variety of dimensional inputs and camera information (e.g., model number, resolution, etc.) and their associated positional parameter outputs. Among the dimensional inputs may be perceived height of the environment (e.g., floor to ceiling), perceived distance of farthest and/or closest point in the view, whether a majority of persons/objects are captured via an overhead view or not, the perceived width of the environment, etc. The machine learning algorithm may use linear/polynomial regression. Accordingly, when the dimensional inputs are camera information are inputted into the machine learning algorithm, updating component440may determine the new positional parameters of the camera and update its records (e.g., store in memory).

At block514, the method500includes mapping a layout of the environment captured in the view of the camera. For example, in an aspect, computer device400, processor405, memory410, layout mapping component415, and/or mapping component445may be configured to or may comprise means for mapping a layout of environment100captured in the view of camera102.

Layout300includes at least one of: a pathway where the plurality of persons can move, a landmark in environment100(e.g., detected using object recognition algorithms), or an ingress/egress point in environment100. In terms of pathways, mapping component445may aggregate each respective path to determine the pathway. For example, as multiple persons walk in environment100, their movement is tracked at represented as vectors. Adding these vectors provides a combinations of movements in areas where walking is permitted. By estimating the width of each person and their motion, the dimensions of each pathway (which may be a combination of physical roads or aisles) may be extrapolated. More specifically, the vectors may be stored in memory410and tagged with an identifier comprising the approximate width of each person associated with the vector.

Ingress/egress points may be identified by mapping component445using a combination of object recognition (to identify doors) and detecting appearance/disappearance of a tracked person in a frame. For example, if a person appears within the frame and disappears within the frame (rather than by exiting from an edge of the frame), mapping component445determines that the person is entering and exiting through a door.

At block516, the method500includes incrementing (i). For example, in an aspect, computer device400, processor405, memory410, layout mapping component415, and/or monitoring component425may be configured to or may comprise means for incrementing (i) by 1.

At block518, the method500includes determining whether (i) is greater than N. For example, in an aspect, computer device400, processor405, memory410, layout mapping component415, and/or monitoring component425may be configured to or may comprise means for determining whether (i) is greater than N. If (i) is not greater than N, there are other persons in the plurality of persons that still have to be monitored and analyzed. Accordingly, method500returns to block506, where the subsequent person (e.g., person106) is identified.

If (i) is greater than N, method500ends. In some aspects, layout mapping component415may generate a graphical user interface on computing device400(e.g., via a peripheral device such as a monitor) that displays layout300(i.e., the output of block514). This enables a user monitoring environment100to see the dimensions of environment100, the pathways used, the ingress/egress points, landmarks, etc., without manually specifying where camera102is placed. In some aspects, layout300may identify routes that are most often used (e.g., by differentiating color of routes) so that the user can view areas of environment100that experience the most foot traffic.

In the initial estimation of the dimensions of environment100and the positional parameters of camera102, the height of person104and movement speed was predetermined. Depending on the real life height and speed of person104, the estimates may be incorrect. Aspects of the present disclosure iteratively update the dimensions of environment100and the positional parameters to ensure that over time, the true dimensions of environment100are determined.

In other words, person104may be in fact 5 feet 6 inches and person106may be 6 feet exactly. The average height between the two persons is 5 feet 9 inches. Accordingly, if the dimensions of environment100are first determined based on the assumption that the height of person104is 5 feet 9 inches and the dimensions of environment100are again determined based on the assumption that the height of person106is 5 feet 9 inches, averaging out both determined dimensions of environment100will in fact yield the true dimensions of environment100. This is a simplistic example in which the heights of two individuals averages to the predetermined height. However, depending on the persons captured by camera102, averaging the dimensions of environment100and the positional parameters of camera102to get their true values may require several persons to be captured.

Thus, after the actions of blocks510and512are performed for the first person identified, blocks510and512for the second person and all subsequently identified persons further comprise averaging the dimensions with the dimensions determined using the movement of the previous person. Suppose that person106also walks along path106. Updating components435and440will re-determine the dimensions of environment100and the positional parameters of camera102based on the assumption that person106is also 5 feet 9 inches. Subsequently, updating component435will set the latest values of the dimensions of environment100as the average values between the dimensions calculated for person104and the dimensions calculated for person106. Updating component440will set the latest values of the positional parameters as the average values between the parameters calculated for person104and the parameters calculated for person106.

It should be noted that although the example of people is given, layout mapping component415may also monitor other moving objects to approximate the dimensions of an environment. For example, if a camera is placed in a parking lot, objects such as cars can be used. Initial estimates of vehicle speed and vehicle dimensions can be applied such that the concept of marking pathways can be implemented with the use of vehicles. Accordingly, the methods discussed in the present disclosure are applicable such that persons are replaced with vehicles or, in some aspects, are considered along with vehicles.

In some aspects, subsequent to a threshold number of persons being analyzed (e.g., 100 individuals), layout mapping component415may update the estimated height of the person and the movement speed based on the updated positional parameters of the camera. For example, the 101th person to be captured by camera102may be assigned his/her own estimated height rather than the predetermined height. This is because the first threshold number of persons are used to calibrate camera102. Assuming that the average height of the 100 individuals is 5 feet 9 inches, the height of the 101th person can be determined based on the conversion between physical distance and pixels of the frame.

FIG. 6is a flowchart illustrating method600of updating the FoV and the focal length of the camera, in accordance with exemplary aspects of the present disclosure.

At block602, the method600includes identifying a curvature in the respective path as captured in the view. For example, in an aspect, computer device400, processor405, memory410, layout mapping component415, and/or identifying component450may be configured to or may comprise means for identifying a curvature in path108as captured in view200.

Under the assumption that people generally walk in straight paths, if path108has a curvature such that the vector describing path108—particularly the second portion of path108—is represented by a plurality of unique vectors, identifying component450may determine that the stream captured by camera102comprises warping. Due to the warping, straight movements in the real physical world have a curvature in the stream.

At block604, the method600includes determining the FoV and the focal length based on a machine learning algorithm that is a function of the updated dimensions and a degree of the curvature/warping. For example, in an aspect, computer device400, processor405, memory410, layout mapping component415, and/or determining component451may be configured to or may comprise means for determining the FoV and the focal length based on a machine learning algorithm that is a function of the updated dimensions and a degree of the curvature.

As mentioned before, the FoV and the focal length may be determined via a machine learning algorithm that receives dimensional inputs (i.e., the updated dimensions). In addition, the machine learning algorithm may receive a degree of curvature/warping, which is determined based on the movement of a person. Based on the assumption that people generally walk in straight paths, if the camera detects a consistent curvature in the paths that people take (e.g., a certain percentage of persons show the same curvature while walking along an aisle in a supermarket—despite the aisle being straight), the curvature may be attributed to the camera. For example, the camera may be in a “wide-angle” mode or “fish-eye” mode. The degree of curvature is a mathematical represent of the magnitude of warping (e.g., in radians/m). The FoV and focal length may thus be determined using a machine learning algorithm configured to receive dimensional inputs and a degree of curvature in order to output the FoV and focal length.