A 3D scanning toolkit to perform operations that include: accessing a first data stream at a client device, wherein the first data stream comprises at least image data; applying a bit mask to the first data stream, the bit mask identifying a portion of the image data; accessing a second data stream at the client device, the second data stream comprising depth data associated with the portion of the image data; generating a point cloud based on the depth data, the point cloud comprising a set of data points that define surface features of an object depicted in the first data stream; and causing display of a visualization of the point cloud upon a presentation of the first data stream at the client device.

TECHNICAL FIELD

Embodiments of the present disclosure relate generally to three-dimensional (3D) modeling, and more particularly, to systems for generating 3D models.

BACKGROUND

3D modeling is the process of developing a mathematical representation of a surface of an object in three dimensions, via specialized sensors and software. 3D models represent the surfaces of objects using a collection of points in 3D space, connected by various geometric entities such as triangles, lines, and curved surfaces.

3D models can be generated by a 3D scanner, which can be based on many different technologies, each with their own limitations, advantages, and costs.

DETAILED DESCRIPTION

As discussed above, 3D modeling is the process of developing a mathematical representation of a surface of an object in three dimensions, via specialized sensors and software. While existing methods of generating 3D models are functionally effective, they are often difficult and inconvenient to apply in a number of use cases. As a result, a more user-friendly approach is needed.

Example embodiments described herein relate to a system that includes a 3D scanning toolkit to perform operations that include: accessing a first data stream at a client device, wherein the first data stream comprises at least image data; applying a bit mask to the first data stream, the bit mask identifying a portion of the image data; accessing a second data stream at the client device, the second data stream comprising depth data associated with the portion of the image data; generating a point cloud based on the depth data, the point cloud comprising a set of data points that define surface features of an object depicted in the first data stream; and causing display of a visualization of the point cloud upon a presentation of the first data stream at the client device.

According to some example embodiments, the first data stream and second data stream accessed at the client device may comprise RGB-D data, wherein each data point comprises an RGB component as well as a depth component. The depth data from the data stream indicates a distance between an image plane and an objected depicted by the data stream, where the image plane is identified as the plane of a display monitor or device users to view an image rendered based on the data stream.

In some example embodiments, responsive to accessing the first data stream at the client device, the system accesses and applies a bit-mask to the first data stream, wherein the bit-mask defines which data points of the data stream to scan for depth data. For example, the bit-mask may specify areas depicted by the data stream “to be scanned,” or “not to be scanned,” (i.e., sets areas depicted by the data stream that are not within the bit-mask to a null value) based on attributes of the data points. As an illustrative example, the data stream may depict a person, and the bit-mask may be configured to mask out everything but the person's head, or even specific portions of the person's head (i.e., just circumference of top of head) such that depth data indicating surface features of the person's head is collected.

In some embodiments, the 3D modeling toolkit may provide an interface to enable a user to provide a selection of one or more bit-masks to be applied to a data stream, wherein each of the one or more bit-masks may correspond with a different object or category. The data stream may comprise image data (e.g., pictures or video) that depicts one or more objects or people. The bit-masks may therefore be organized based on object categories, or measurement types. For example, a bit mask may be associated with a measurement category for “helmet,” or “glasses,” wherein the corresponding bit-masks mask out the pixels not needed for the measurements. Accordingly, a bit-mask associated with the “helmet” measurement category may filter out everything in the image but a person's head (or specific portions of a person's head). In further embodiments, the 3D modeling toolkit may perform one or more image recognition techniques to identify objects depicted in the image data of a data-stream in order to automatically select one or more bit-masks to present to a user of the 3D modeling toolkit as recommendations.

Based on the bit-mask applied to the first data stream, the system accesses a portion of a second data stream that comprises depth data, wherein the portion of the second data stream is based on the bit-mask. In such embodiments, the 3D modeling toolkit may access portions of the second data stream that correspond with the pixels indicated based on the bit-mask applied to the first data stream. For example, the bit-mask may assign a binary value to each pixel of an image generated based on the image data, to indicate if the system “should,” or “should not” access the second data stream to scan a particular area depicted by the image data of the first data stream.

Based on the depth data of the second data stream, the system generates a point cloud. As discussed herein, a point cloud is a set of data points in a space which depicts the external surfaces of objects. In some example embodiments, the point cloud may be converted into a 3D model. For example, the point cloud may be converted into a polygon mesh model, a triangle mesh model, a non-uniform rational basis spline (NURBS) surface model, or a CAD model through one or more surface reconstruction techniques.

The system causes display of a visualization of the point cloud within a presentation of the image data from the first data stream at the client device. The visualization may for example be based on the 3D model generated based on the point cloud.

According to certain embodiments, the system saves the 3D model generated based on the point cloud at a memory location at the client device, or in some embodiments at a remote database. For example, the system may present an option to save the 3D model in the presentation of the image data at the client device.

In some example embodiments, the 3D modeling toolkit may provide one or more interfaces to generate training data for a machine learned model. For example, the 3D modeling toolkit may access a memory repository that comprises one or more 3D models generated based on point clouds and provide an interface to display presentations of the 3D models at a client device. A user of the client device may provide semantic labels to be applied to the 3D models through the one or more interfaces. The labeled 3D models may then be utilized to train a machine learned model.

For example, a machine learned model may be fit on a training dataset, wherein the training dataset is generated based on the point clouds collected by the 3D modeling toolkit. The machine learned model may then be trained using a supervised learning method.

FIG. 1is a block diagram showing an example modeling system100for exchanging data over a network. The modeling system100include one or more client devices102which host a number of applications including a client application104. Each client application104is communicatively coupled to other instances of the client application104and a server system108via a network106(e.g., the Internet).

Accordingly, each client application104is able to communicate and exchange data with another client application104and with the server system108via the network106. The data exchanged between client applications104, and between a client application104and the server system108, includes functions (e.g., commands to invoke functions) as well as payload data (e.g., text, audio, video or other multimedia data).

The server system108provides server-side functionality via the network106to a particular client application104. While certain functions of the modeling system100are described herein as being performed by either a client application104or by the server system108, it will be appreciated that the location of certain functionality either within the client application104or the server system108is a design choice. For example, it may be technically preferable to initially deploy certain technology and functionality within the server system108, but to later migrate this technology and functionality to the client application104where a client device102has a sufficient processing capacity.

The server system108supports various services and operations that are provided to the client application104. Such operations include transmitting data to, receiving data from, and processing data generated by the client application104. In some embodiments, this data includes, image data, Red-blue-green (RBG) data, depth data, inertial measurement unit (IMU) data, client device information, geolocation information, as examples. In other embodiments, other data is used. Data exchanges within the modeling system100are invoked and controlled through functions available via GUIs of the client application104.

Dealing specifically with the Application Program Interface (API) server110, this server receives and transmits data between the client device102and the application server112. Specifically, the Application Program Interface (API) server110provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the client application104in order to invoke functionality of the application server112. The Application Program Interface (API) server110exposes various functions supported by the application server112, including account registration, login functionality, the sending of messages or content, via the application server112, from a particular client application104to another client application104, the sending of media files (e.g., images or video) from a client application104to the server application114, and for possible access by another client application104, opening and application event (e.g., relating to the client application104).

The application server112hosts a number of applications and subsystems, including a server application114, an image processing system116, and a 3D modeling toolkit124. The server application114implements a number of image processing technologies and functions, particularly related to the aggregation and other processing of content (e.g., image data) received from multiple instances of the client application104. As will be described in further detail, the image data from multiple sources may be aggregated into collections of content. These collections are then made available, by the server application114, to the client application104. Other processor and memory intensive processing of data may also be performed server-side by the messaging server application114, in view of the hardware requirements for such processing.

The application server112also includes an image processing system116that is dedicated to performing various image processing operations, typically with respect to images or video received from one or more client devices102at the messaging server application114.

The application server112is communicatively coupled to a database server118, which facilitates access to a database120in which is stored data associated with image data processed by the messaging server application114.

FIG. 2is a block diagram illustrating components of the 3D modeling toolkit124that configure the 3D modeling toolkit124to generate a 3D model based on a point cloud, according to certain example embodiments.

The 3D modeling toolkit124is shown as including an image module202, a bit mask module204, a depth data module206, a 3D model module208, and an analysis module210, all configured to communicate with each other (e.g., via a bus, shared memory, or a switch). Any one or more of these modules may be implemented using one or more processors212(e.g., by configuring such one or more processors to perform functions described for that module) and hence may include one or more of the processors212.

Any one or more of the modules described may be implemented using hardware alone (e.g., one or more of the processors212of a machine) or a combination of hardware and software. For example, any module described of the 3D modeling toolkit124may physically include an arrangement of one or more of the processors212(e.g., a subset of or among the one or more processors of the machine) configured to perform the operations described herein for that module. As another example, any module of the 3D modeling toolkit124may include software, hardware, or both, that configure an arrangement of one or more processors212(e.g., among the one or more processors of the machine) to perform the operations described herein for that module. Accordingly, different modules of the 3D modeling toolkit124may include and configure different arrangements of such processors212or a single arrangement of such processors212at different points in time. Moreover, any two or more modules of the 3D modeling toolkit124may be combined into a single module, and the functions described herein for a single module may be subdivided among multiple modules. Furthermore, according to various example embodiments, modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

FIG. 3is a flowchart illustrating a method300for generating and causing display of a 3D model at a client device102, according to certain example embodiments. Operations of the method300may be performed by the modules described above with respect toFIG. 2. As shown inFIG. 3, the method300includes one or more operations302,304,306, and308.

At operation302, the image module202accesses a first data stream at the client device102, wherein the first data stream comprises image data. For example, the image data may include RGB data.

At operation304, the bit mask module204applies a bit mask to the first data stream, wherein the bit mask identifies a portion of the image data. In some embodiments, the bit mask module204may access the bit mask based on an input received from the client device102or based on attributes of the image data from the first data stream. For example, the bit mask may be selected from among a plurality of bit masks, wherein each bit mask among the plurality of bit masks is configured based on features of image data.

In some embodiments, the bit mask module204may apply a machine learned model to identify the portion of the image data. For example, as will be discussed in more detail in the method400ofFIG. 4, the bit mask module204may access a machine learned model based on attributes of the image data, wherein the machine learned model is trained to apply one or more semantic labels to the image data, wherein the one or more semantic labels may indicate regions within the image data to scan or not.

For example, the bit mask may assign a binary pixel value to areas in the image data based on features of the image data. By doing so, some areas in the image data (i.e., those areas assigned a 0 pixel value) may be “masked,” indicating that those areas are not to be scanned, while other areas (i.e., those areas assigned a 1 pixel values) are scanned for depth data.

At operation306, the depth data module206accesses a second data stream at the client device102, wherein the second data stream comprises depth data associated with the portion of the image data identified based on the bit mask applied to the image data.

At operation308, the depth data module206generates a point cloud based on the depth data, wherein the point cloud comprises a set of data points that define surface features of an object depicted in the first data stream. At operation310, the 3D modeling module208generates and causes display of a visualization of the point cloud upon a presentation of the first data stream at the client device102.

FIG. 4is a flowchart illustrating a method400for preparing a training data set for a machine learned model, according to certain example embodiments. Operations of the method400may be performed by the modules described above with respect toFIG. 2. As shown inFIG. 4, the method400includes one or more operations402,404,406, and408, that may be performed as a part of (e.g., a subroutine, or subsequent to) the method300depicted inFIG. 3.

According to certain example embodiments, subsequent to operation308of the method300, wherein a point cloud is generated by the depth module206, at operation402the 3D modeling toolkit124receives an input that selects a subset of the set of data points of the point cloud. For example, the 3D modeling toolkit124may cause display of an interface to receive inputs selecting the subset of the set of data points, wherein the inputs may for example include an input that “paints” the subset of the set of data points with a cursor, or in some embodiments through a tactile input.

At operation404the 3D modeling toolkit124applies a label to the subset of the set of data point identified based on the input. The label may include a semantic label, or a classification. For example, semantic labeling features may for example include: contextual features that correspond with a physical object, location, or surface; analogical features that reference some other known category or class; visual features that define visual or graphical properties of a surface or object; as well as material parameters that define properties of a surface or object and which may include a “roughness value,” a “metallic value,” a “specular value,” and a “base color value.”

At operation406, the 3D modeling toolkit124generates a training dataset based on the label and the subset of the set of data points identified based on the input.

At operation408, the 3D modeling toolkit124fits a machine learned model to the training dataset. Accordingly, the machine learned model may be trained to apply semantic labels to portions of image data, wherein the semantic labels include bit-mask values (i.e., binary values indicating to scan or not scan).

FIG. 5is a flowchart illustrating a method500for presenting a value based on a point cloud, according to certain example embodiments. Operations of the method500may be performed by the modules described above with respect toFIG. 2. As shown inFIG. 5, the method500includes one or more operations502,504,506,508,510,512, and514, that may be performed as a part of (e.g., a subroutine, or subsequent to) the method300depicted inFIG. 3.

At operation502, the analysis module210accesses the point cloud generated by the depth data module206at operation308of the method300. At operation504, the analysis module210identifies a plurality of landmarks based on the point cloud.

Responsive to identifying the plurality of landmarks, at operation506the analysis module210determines a classification associated with the landmarks, and at operation508, causes the 3D model module208to retrieve a 3D model associated with the classification and the plurality of landmarks. For example, the database120may comprise a collection of 3D models accessible by the 3D modeling toolkit124, wherein each 3D model among the collection of 3D models is associated with one or more landmarks.

At operation510, the 3D model module208applies the 3D model to a position in a 3D space relative to the point cloud based on at least the plurality of landmarks of the point cloud.

At operation512, the analysis module210generates a value based on the position of the 3D model relative to the point cloud, and at operation514causes display of the value at the client device102.

As an illustrative example from a user perspective, the point cloud may depict a 3D representation of a human head. The analysis module210may analyze the point cloud to detect key landmarks (e.g., facial landmarks, etc.) for alignment and classification, and to fill missing portions of the point cloud by using machine learning.

Responsive to analyzing the point cloud and identifying the landmarks, the 3D model module208retrieves a 3D model of a helmet, and positions the 3D model of the helmet at a position relative to the point cloud that depicts the human head in order to calculate distances between the landmarks of the point cloud and the 3D model according to a geometric algorithm. The 3D model of the helmet and the point cloud are analyzed further in order to estimate sizing (i.e., a value), which can then be presented at a client device.

In some embodiments, the analysis module210may filter a collection based on the value. For example, the collection may comprise a plurality of objects with associated size values. The analysis module210may access the collection and filter the collection based on the value generated based on the position of the 3D model relative to the point cloud. In some embodiments, the filtered collection may then be presented at the client device102.

FIG. 6is an interface flow diagram600illustrating interfaces presented by the 3D modeling toolkit124, according to certain example embodiments, and as discussed in the method300depicted inFIG. 3.

Interface602depicts an interface to initiate a 3D scan. For example, a user of the 3D scanning toolkit124may provide an input through the interface element608that causes one or more modules of the 3D scanning toolkit124to initiate a 3D scan.

Interface604depicts a presentation of depth data610based on a second data stream, as discussed in operation306of the method300. The depth data provides an indication of a distance of any given point to a reference position (i.e., a camera of the client device102).

Interface606depicts a 3D model612generated based on a first data stream (i.e., image data), and a second data stream (i.e., depth data) presented at a client device102. According to certain embodiments, the 3D modeling toolkit124may save the 3D model612at a memory location at the client device102, or in some embodiments at a remote database such as the database120.

FIG. 7is a diagram700depicting a labeled point cloud702, according to certain example embodiments, and as discussed in the method400depicted inFIG. 4. As seen in the diagram700, a user of the 3D modeling toolkit124may provide input applying one or more labels to the point cloud702.

As seen in the diagram700, and as discussed in operation308of the method300depicted inFIG. 3, the depth data module206generates a point cloud (i.e., the point cloud702) based on depth data, wherein the point cloud702comprises a set of data points that define surface features of an object depicted in a first data stream. For example, each data point of the point cloud702(e.g., data point706) comprises data attributes that include location data as well as depth data that identify a position of the data point in a space.

In some example embodiments, the 3D modeling toolkit124may provide one or more interfaces to generate training data for a machine learned model. For example, a user of the client device102may provide inputs that select one or more data points from among the plurality of data points that make up the point cloud702, to apply one or more semantic labels to the one or more data points. As seen in the diagram700, the labeled points704may be presented in a different color or pattern from the unlabeled points of the point cloud702.

FIG. 8is a diagram800depicting a 3D model802retrieved based on a point cloud848, according to certain example embodiments. As seen in the diagram800, the point cloud804defines a set of surface features of an object (i.e., a face). As discussed in operation502of the method500depicted inFIG. 5, the analysis module210accesses the point cloud804generated by the depth data module206and identifies a plurality of landmarks based on the point cloud804.

The analysis module210determines a classification associated with the landmarks defined by the point cloud804and causes the 3D model module208to retrieve a 3D model802from a collection of 3D models, based on at least the classification associated with the plurality of landmarks. The 3D model802may then be presented at a position among the presentation of the point cloud804at the client device102.

Software Architecture

FIG. 9is a block diagram illustrating an example software architecture906, which may be used in conjunction with various hardware architectures herein described.FIG. 9is a non-limiting example of a software architecture and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. The software architecture906may execute on hardware such as the machine900ofFIG. 9that includes, among other things, processors904, memory914, and I/O components918. A representative hardware layer952is illustrated and can represent, for example, the machine1000ofFIG. 10. The representative hardware layer952includes a processing unit954having associated executable instructions904. Executable instructions904represent the executable instructions of the software architecture906, including implementation of the methods, components and so forth described herein. The hardware layer952also includes memory and/or storage modules memory/storage956, which also have executable instructions904. The hardware layer952may also comprise other hardware958.

In the example architecture ofFIG. 9, the software architecture906may be conceptualized as a stack of layers where each layer provides particular functionality. For example, the software architecture906may include layers such as an operating system902, libraries920, applications916and a presentation layer914. Operationally, the applications916and/or other components within the layers may invoke application programming interface (API) API calls908through the software stack and receive a response as in response to the API calls908. The layers illustrated are representative in nature and not all software architectures have all layers. For example, some mobile or special purpose operating systems may not provide a frameworks/middleware918, while others may provide such a layer. Other software architectures may include additional or different layers.

The operating system902may manage hardware resources and provide common services. The operating system902may include, for example, a kernel922, services924and drivers926. The kernel922may act as an abstraction layer between the hardware and the other software layers. For example, the kernel922may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, and so on. The services924may provide other common services for the other software layers. The drivers926are responsible for controlling or interfacing with the underlying hardware. For instance, the drivers926include display drivers, camera drivers, Bluetooth® drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi® drivers, audio drivers, power management drivers, and so forth depending on the hardware configuration.

The libraries920provide a common infrastructure that is used by the applications916and/or other components and/or layers. The libraries920provide functionality that allows other software components to perform tasks in an easier fashion than to interface directly with the underlying operating system902functionality (e.g., kernel922, services924and/or drivers926). The libraries920may include system libraries944(e.g., C standard library) that may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. In addition, the libraries920may include API libraries946such as media libraries (e.g., libraries to support presentation and manipulation of various media format such as MPREG4, H.264, MP3, AAC, AMR, JPG, PNG), graphics libraries (e.g., an OpenGL framework that may be used to render 2D and 3D in a graphic content on a display), database libraries (e.g., SQLite that may provide various relational database functions), web libraries (e.g., WebKit that may provide web browsing functionality), and the like. The libraries920may also include a wide variety of other libraries948to provide many other APIs to the applications916and other software components/modules.

The frameworks/middleware918(also sometimes referred to as middleware) provide a higher-level common infrastructure that may be used by the applications916and/or other software components/modules. For example, the frameworks/middleware918may provide various graphic user interface (GUI) functions, high-level resource management, high-level location services, and so forth. The frameworks/middleware918may provide a broad spectrum of other APIs that may be utilized by the applications916and/or other software components/modules, some of which may be specific to a particular operating system902or platform.

The applications916include built-in applications938and/or third-party applications940. Examples of representative built-in applications938may include, but are not limited to, a contacts application, a browser application, a book reader application, a location application, a media application, a messaging application, and/or a game application. Third-party applications940may include an application developed using the ANDROID™ or IOS™ software development kit (SDK) by an entity other than the vendor of the particular platform, and may be mobile software running on a mobile operating system such as IOS™, ANDROID™, WINDOWS® Phone, or other mobile operating systems. The third-party applications940may invoke the API calls908provided by the mobile operating system (such as operating system902) to facilitate functionality described herein.

The applications916may use built in operating system functions (e.g., kernel922, services924and/or drivers926), libraries920, and frameworks/middleware918to create user interfaces to interact with users of the system. Alternatively, or additionally, in some systems interactions with a user may occur through a presentation layer, such as presentation layer914. In these systems, the application/component “logic” can be separated from the aspects of the application/component that interact with a user.

The machine1000may include processors1004, memory memory/storage1006, and I/O components1018, which may be configured to communicate with each other such as via a bus1002. The memory/storage1006may include a memory1014, such as a main memory, or other memory storage, and a storage unit1016, both accessible to the processors1004such as via the bus1002. The storage unit1016and memory1014store the instructions1010embodying any one or more of the methodologies or functions described herein. The instructions1010may also reside, completely or partially, within the memory1014, within the storage unit1016, within at least one of the processors1004(e.g., within the processor's cache memory), or any suitable combination thereof, during execution thereof by the machine1000. Accordingly, the memory1014, the storage unit1016, and the memory of processors1004are examples of machine-readable media.

The I/O components1018may include a wide variety of components to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so on. The specific I/O components1018that are included in a particular machine1000will depend on the type of machine. For example, portable machines such as mobile phones will likely include a touch input device or other such input mechanisms, while a headless server machine will likely not include such a touch input device. It will be appreciated that the I/O components1018may include many other components that are not shown inFIG. 10. The I/O components1018are grouped according to functionality merely for simplifying the following discussion and the grouping is in no way limiting. In various example embodiments, the I/O components1018may include output components1026and input components1028. The output components1026may include visual components (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., a vibratory motor, resistance mechanisms), other signal generators, and so forth. The input components1028may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instrument), tactile input components (e.g., a physical button, a touch screen that provides location and/or force of touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like.

Communication may be implemented using a wide variety of technologies. The I/O components1018may include communication components1040operable to couple the machine1000to a network1032or devices1020via coupling1022and coupling1024respectively. For example, the communication components1040may include a network interface component or other suitable device to interface with the network1032. In further examples, communication components1040may include wired communication components, wireless communication components, cellular communication components, Near Field Communication (NFC) components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components to provide communication via other modalities. The devices1020may be another machine or any of a wide variety of peripheral devices (e.g., a peripheral device coupled via a Universal Serial Bus (USB)).

GLOSSARY

“3D RECONSTRUCTION” in this context refers to a process of building a 3D model using multiple pieces of partial information about a subject.

“3D SCAN” in this context refers to the result of a 3D reconstruction.

“SIMULTANEOUS LOCATION AND MAPPING (SLAM)” in this context refers to a method of building a map or model of an unknown scene or subject while simultaneously keeping track of a device position within an environment.

“DEPTH FRAME” in this context refers to a snapshot in time of depth values from a sensor, arranged in a 2D grid, like an RGB camera frame. In certain embodiments the depth values are the distance in meters from a device to a subject.

“POINT CLOUD” in this context refers to and unordered array of points in 3D, wherein each point has an XYZ position, a color, a normal (which is a vector indicating the point's orientation), and other information.

“MESH” in this context refers to a collection of triangles.