Patent Description:
While the invention is set out in the appended claims, the following description is provided for improving the understanding of the invention as claimed. Additive manufacturing may be used to manufacture 3D objects. Three-dimensional (3D) printing is an example of additive manufacturing. Some examples of 3D printing may selectively deposit agents (e.g., droplets) at a pixel level to enable control over voxel-level energy deposition. For instance, thermal energy may be projected over material in a build area, where a phase change and solidification in the material may occur depending on the voxels where the agents are deposited.

Predicting transient thermal behavior with print process resolution (e.g., voxel-by-voxel in space and/or layer-by-layer in time) may be used to improve offline print tuning and/or online printing control. However, it is difficult to derive a quantitative model for predicting transient thermal behavior due to a lack of quantitative knowledge in terms of how material behaves.

Thermal behavior may be mapped as a thermal image. A thermal image is a set of data indicating temperature (or thermal energy) in an area. Thermal mapping includes obtaining (e.g., sensing and/or capturing) a thermal image or images and/or calculating (e.g., predicting) a thermal image or images. In some examples, a machine learning model (e.g., neural network or networks) may be used to calculate predicted thermal images. A predicted thermal image is a thermal image that is calculated using a machine learning model. For instance, the neural network or networks may utilize a contone map or maps (e.g., voxel-level machine instructions that dictate the placement, quantity, and/or timing of an agent or agents in a build area) and/or a thermal image or images to predict a thermal image.

A captured thermal image is a thermal image that is sensed or captured with a sensor. Sensors for capturing thermal images may be limited in resolution. For example, a built-in sensor in an additive manufacturing device may provide relatively low resolution (e.g., <NUM> x <NUM> pixels, <NUM> x <NUM> pixels, etc.) for online (e.g., run-time) thermal imaging. It may be beneficial to utilize a low-resolution thermal image sensor built-in to an additive manufacturing device due to the expense, size, and/or other considerations that may keep a high-resolution sensor from being utilized.

Low resolution thermal imaging may be inadequate to support voxel level thermal prediction in some approaches. For example, some approaches (e.g., some interpolation-based approaches, statistical approaches, and/or example-based approaches) for upscaling are inadequate to upscale the image resolution by a relatively large factor (e.g., <NUM>, <NUM>, etc.) accurately. For instance, some approaches may offer limited upscaling by a factor of <NUM> to <NUM> and may lack accuracy. Some of these approaches for increasing image resolution are based on visual spectrum images, and may involve a one-to-one mapping. Because fusing layer thermal behavior follows physical laws, there is additional potential for increasing the resolution of thermal images. However, many approaches are not designed for thermal images, and are not designed to leverage additional useful information and achieve physically consistent thermal sensing enhancement.

Some examples of the techniques described herein may include a deep neural network based practical model training approach that can achieve voxel-level thermal prediction with low-resolution thermal sensing and a contone map or maps as input. In some examples, thermal image prediction that is approximately <NUM> times the resolution of thermal sensing can be achieved (e.g., from <NUM> x <NUM> pixels or <NUM> x <NUM> pixels to <NUM> x <NUM> pixels). Missing details may be inferred from additional information (e.g., contone maps). Accordingly, some examples of the techniques described herein may significantly exceed other approaches. Some examples may enable online in-situ voxel-level thermal image prediction and/or online closed-loop feedback control.

It should be noted that while plastics may be utilized as a way to illustrate some of the approaches described herein, the techniques described herein may be applied to some examples of additive manufacturing. Some additive manufacturing techniques may be powder-based and driven by powder fusion. Some examples of the approaches described herein may be applied to powder bed fusion-based additive manufacturing, such as Selective Laser Melting (SLM), Selective Laser Sintering (SLS), etc. Some examples of the approaches described herein may be applied to additive manufacturing where agents carried by droplets are utilized for voxel-level thermal modulation. It should be noted that agents may or may not be utilized in some examples. It should also be noted that while some resolutions are described herein as examples, the techniques described herein may be applied for different resolutions.

As used herein, the term "voxel" and variations thereof refers to a "thermal voxel. " In some examples, the size of a thermal voxel may be defined as a minimum that is thermally meaningful (e.g., larger than <NUM> microns or <NUM> dots per inch (dpi)). An example of voxel size is <NUM> millimeters (mm)/<NUM> ≈ <NUM> microns for <NUM> dots per inch (dpi). A maximum voxel size may be approximately <NUM> microns or <NUM> dpi. The term "voxel level" and variations thereof may refer to a resolution, scale, or density corresponding to voxel size. The term "low resolution" and variations thereof refer to a resolution, scale, or density that is lower than that of a voxel level. For example, a low resolution is lower than a voxel-level resolution. Low-resolution thermal imaging may depend on the pixel resolution in a manufacturing device (e.g., machine, printer, etc.). For example, pixel size in low resolution thermal imaging may range from <NUM> to <NUM>. While an example of low-resolution size is given, other low-resolution sizes may be utilized.

Throughout the drawings, identical reference numbers may designate similar, but not necessarily identical, elements.

<FIG> is a simplified isometric view of an example of a 3D printing device <NUM> that may be used in an example of thermal mapping. The 3D printing device <NUM> includes a controller <NUM>, a data store <NUM>, a build area <NUM>, a print head <NUM>, a fusing agent container <NUM>, a detailing agent container <NUM>, a roller <NUM>, a material container <NUM>, a thermal projector <NUM>, and/or a thermal sensor <NUM>. The example of a 3D printing device <NUM> in <FIG> may include additional components that are not shown, and some of the components described may be removed and/or modified without departing from the scope of the 3D printing device <NUM> in this disclosure. The components of the 3D printing device <NUM> may not be drawn to scale, and thus, may have a size and/or configuration different than what is shown.

In the example of <FIG>, the 3D printing device <NUM> includes a fusing agent container <NUM>, fusing agent <NUM>, a detailing agent container <NUM>, detailing agent <NUM>, a material container <NUM>, and material <NUM>. In other examples, the 3D printing device <NUM> may include more or fewer containers, agents, hoppers, and/or materials. The material container <NUM> is a container that stores material <NUM> that may be applied (e.g., spread) onto the build area <NUM> by the roller <NUM> for 3D printing. The fusing agent container <NUM> is a container that stores a fusing agent <NUM>. The fusing agent <NUM> is a substance (e.g., liquid, powder, etc.) that controls intake thermal intensity. For example, the fusing agent <NUM> may be selectively applied to cause applied material <NUM> to change phase with heat applied from the thermal projector <NUM> and/or to fuse with another layer of material <NUM>. For instance, areas of material <NUM> where the fusing agent <NUM> has been applied may eventually solidify into the object being printed. The detailing agent <NUM> is a substance (e.g., liquid, powder, etc.) that controls outtake thermal intensity. For example, the detailing agent <NUM> may be selectively applied to detail edges of the object being printed.

The build area <NUM> is an area (e.g., surface) on which additive manufacturing may be performed. In some configurations, the build area <NUM> may be the base of a "build volume," which may include a volume above the base. As used herein, the term "build area" may refer to the base of a build volume and/or another portion (e.g., another plane above the base) of the build volume.

The roller <NUM> is a device for applying material <NUM> to the build area <NUM>. In order to print a 3D object, the roller <NUM> may successively apply (e.g., spread) material <NUM> (e.g., a powder) and the print head <NUM> may successively apply and/or deliver fusing agent <NUM> and/or detailing agent <NUM>. The thermal projector <NUM> is a device that delivers energy (e.g., thermal energy, heat, etc.) to the material <NUM>, fusing agent <NUM>, and/or detailing agent <NUM> in the build area <NUM>. For example, fusing agent <NUM> may be applied on a material <NUM> layer where particles (of the material <NUM>) are meant to fuse together. The detailing agent <NUM> may be applied to modify fusing and create fine detail and/or smooth surfaces. The areas exposed to energy (e.g., thermal energy from the thermal projector <NUM>) and reactions between the agents (e.g., fusing agent <NUM> and detailing agent <NUM>) and the material <NUM> may cause the material <NUM> to selectively fuse together to form the object.

The print head <NUM> is a device to apply a substance or substances (e.g., fusing agent <NUM> and/or detailing agent <NUM>). The print head <NUM> may be, for instance, a thermal inkjet print head, a piezoelectric print head, etc. The print head <NUM> may include a nozzle or nozzles (not shown) through which the fusing agent <NUM> and/or detailing agent <NUM> are extruded. In some examples, the print head <NUM> may span a dimension of the build area <NUM>. Although a single print head <NUM> is depicted, multiple print heads <NUM> may be used that span a dimension of the build area <NUM>. Additionally, a print head or heads <NUM> may be positioned in a print bar or bars. The print head <NUM> may be attached to a carriage (not shown in <FIG>). The carriage may move the print head <NUM> over the build area <NUM> in a dimension or dimensions.

The material <NUM> is a substance (e.g., powder) for manufacturing objects. The material <NUM> may be moved (e.g., scooped, lifted, and/or extruded, etc.) from the material container <NUM>, and the roller <NUM> may apply (e.g., spread) the material <NUM> onto the build area <NUM> (on top of a current layer, for instance). In some examples, the roller <NUM> may span a dimension of the build area <NUM> (e.g., the same dimension as the print head <NUM> or a different dimension than the print head <NUM>). Although a roller <NUM> is depicted, other means may be utilized to apply the material <NUM> to the build area <NUM>. In some examples, the roller <NUM> may be attached to a carriage (not shown in <FIG>). The carriage may move the roller <NUM> over the build area <NUM> in a dimension or dimensions. In some implementations, multiple material containers <NUM> may be utilized. For example, two material containers <NUM> may be implemented on opposite sides of the build area <NUM>, which may allow material <NUM> to be spread by the roller <NUM> in two directions.

In some examples, the thermal projector <NUM> may span a dimension of the build area <NUM>. Although one thermal projector <NUM> is depicted, multiple thermal projectors <NUM> may be used that span a dimension of the build area <NUM>. Additionally, a thermal projector or projectors <NUM> may be positioned in a print bar or bars. The thermal projector <NUM> may be attached to a carriage (not shown in <FIG>). The carriage may move the thermal projector <NUM> over the build area <NUM> in a dimension or dimensions.

In some examples, each of the print head <NUM>, roller <NUM>, and thermal projector <NUM> may be housed separately and/or may move independently. In some examples, two or more of the print head <NUM>, roller <NUM>, and thermal projector <NUM> may be housed together and/or may move together. In one example, the print head <NUM> and the thermal projector <NUM> may be housed in a print bar spanning one dimension of the build area <NUM>, while the roller <NUM> may be housed in a carriage spanning another dimension of the build area <NUM>. For instance, the roller <NUM> may apply a layer of material <NUM> in a pass over the build area <NUM>, which may be followed by a pass or passes of the print head <NUM> and thermal projector <NUM> over the build area <NUM>.

The controller <NUM> is a computing device, a semiconductor-based microprocessor, a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Field-Programmable Gate Array (FPGA), an application-specific integrated circuit (ASIC), and/or other hardware device. The controller <NUM> may be connected to other components of the 3D printing device <NUM> via communication lines (not shown).

The controller <NUM> may control actuators (not shown) to control operations of the components of the 3D printing device <NUM>. For example, the controller <NUM> may control an actuator or actuators that control movement of the print head <NUM> (along the x-, y-, and/or z-axes), actuator or actuators that control movement of the roller <NUM> (along the x-, y-, and/or z-axes), and/or actuator or actuators that control movement of the thermal projector <NUM> (along the x-, y-, and/or z-axes). The controller <NUM> may also control the actuator or actuators that control the amounts (e.g., proportions) of fusing agent <NUM> and/or detailing agent <NUM> to be deposited by the print head <NUM> from the fusing agent container <NUM> and/or detailing agent container <NUM>. In some examples, the controller <NUM> may control an actuator or actuators that raise and lower build area <NUM> along the z-axis.

The controller <NUM> may communicate with a data store <NUM>. The data store <NUM> may include machine-readable instructions that cause the controller <NUM> to control the supply of material <NUM>, to control the supply of fusing agent <NUM> and/or detailing agent <NUM> to the print head <NUM>, to control movement of the print head <NUM>, to control movement of the roller <NUM>, and/or to control movement of the thermal projector <NUM>.

In some examples, the controller <NUM> may control the roller <NUM>, the print head <NUM>, and/or the thermal projector <NUM> to print a 3D object based on a 3D model. For instance, the controller <NUM> may utilize a contone map or maps that are based on the 3D model to control the print head <NUM>. A contone map is a set of data indicating a location or locations (e.g., areas) for printing a substance (e.g., fusing agent <NUM> or detailing agent <NUM>). In some examples, a contone map may include or indicate machine instructions (e.g., voxel-level machine instructions) for printing a substance. For example, a fusing agent contone map indicates coordinates and/or an amount for printing the fusing agent <NUM>. In an example, a detailing agent contone map indicates coordinates and/or an amount for printing the detailing agent <NUM>. In some examples, a contone map may correspond to a two-dimensional (2D) layer (e.g., 2D slice, 2D cross-section, etc.) of the 3D model. For instance, a 3D model may be processed to produce a plurality of contone maps corresponding to a plurality of layers of the 3D model. In some examples, a contone map may be expressed as a 2D grid of values, where each value may indicate whether to print an agent and/or an amount of agent at the location on the 2D grid. For instance, the location of a value in the 2D grid may correspond to a location in the build area <NUM> (e.g., a location (x, y) of a particular level (z) at or above the build area <NUM>). In some examples, a contone map may be a compressed version of the aforementioned 2D grid or array (e.g., a quadtree).

The data store <NUM> is a machine-readable storage medium. Machine-readable storage is any electronic, magnetic, optical, or other physical storage device that stores executable instructions and/or data. A machine-readable storage medium may be, for example, Random Access Memory (RAM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a storage drive, an optical disc, and the like. A machine-readable storage medium may be encoded with executable instructions for controlling the 3D printing device <NUM>. A computer-readable medium is an example of a machine-readable storage medium that is readable by a processor or computer.

The thermal sensor <NUM> is a device that senses or captures thermal data. The thermal sensor <NUM> may be integrated into, mounted in, and/or otherwise included in a machine (e.g., printer). In some examples, the thermal sensor <NUM> may capture thermal images of the build area <NUM>. For instance, the thermal sensor <NUM> may be an infrared thermal sensor (e.g., camera) that captures thermal images of the build area <NUM> (e.g., applied material in the build area <NUM>). In some examples, the thermal sensor <NUM> may capture thermal images during manufacturing (e.g., printing). For example, the thermal sensor <NUM> may capture thermal images online and/or in real-time.

A thermal image is a set of data indicating temperature (or thermal energy) in an area. A thermal image may be captured (e.g., sensed) from a thermal sensor <NUM> or may be calculated (e.g., predicted). For example, the thermal sensor <NUM> may capture a thermal image of a layer to produce a captured thermal image.

In some examples, a captured thermal image may be a two-dimensional (2D) grid of sensed temperatures (or thermal energy). In some examples, each location in the 2D grid may correspond to a location in the build area <NUM> (e.g., a location (x, y) of a particular level (z) at or above the build area <NUM>). The thermal image or images may indicate thermal variation (e.g., temperature variation) over the build area <NUM>. For example, thermal sensing over the build area <NUM> may indicate (e.g., capture and encapsulate) environmental complexity and heterogeneous thermal diffusivity. In some approaches, the thermal image or images may be transformed to align with a contone map or contone maps (e.g., registered with the contone map or maps).

In some examples, the controller <NUM> may receive a captured thermal image of a layer from the thermal sensor <NUM>. For example, the controller <NUM> may command the thermal sensor <NUM> to capture a thermal image and/or may receive a captured thermal image from the thermal sensor <NUM>. In some examples, the thermal sensor <NUM> may capture a thermal image for each layer of an object being manufactured. The captured thermal image is at a resolution. The resolution of the captured thermal image is lower than a voxel-level resolution. For example, the resolution of the captured thermal image may be at a low-resolution. Examples of low-resolution include <NUM> x <NUM> pixels and <NUM> x <NUM> pixels. Each captured thermal image may be stored as thermal image data <NUM> in the data store <NUM>.

In some examples, the data store <NUM> may store neural network data <NUM>, thermal image data <NUM>, and/or enhanced thermal image data <NUM>. The neural network data <NUM> includes data defining a neural network or neural networks. For instance, the neural network data <NUM> may define a node or nodes, a connection or connections between nodes, a network layer or network layers, and/or a neural network or neural networks. Examples of neural networks include convolutional neural networks (CNNs) (e.g., basic CNN, deconvolutional neural network, inception module, residual neural network, etc.) and recurrent neural networks (RNNs) (e.g., basic RNN, multi-layer RNN, bidirectional RNN, fused RNN, clockwork RNN, etc.). Some approaches may utilize a variant or variants of RNN (e.g., Long Short Term Memory Unit (LSTM), peephole LSTM, no input gate (NIG), no forget gate (NFG), no output gate (NOG), no input activation function (NIAF), no output activation function (NOAF), no peepholes (NP), coupled input and forget gate (CIFG), full gate recurrence (FGR), gated recurrent unit (GRU), etc.). Different depths of a neural network or neural networks may be utilized.

In some examples, the controller <NUM> uses the neural network or networks (defined by the neural network data <NUM>) to predict thermal images. For example, the controller <NUM> may calculate (e.g., predict), using a neural network or a plurality of neural networks, a predicted thermal image of a layer based on a captured thermal image or a plurality of captured thermal images and a contone map or a plurality of contone maps (e.g., a fusing contone map and a detailing contone map). The contone map or maps may be utilized as inputs to the neural network or networks. For instance, a voxel-level contone map or maps may be used in some approaches because the contone map or maps may enable voxel-level energy control and/or may provide information to increase the resolution of the predicted thermal image relative to the resolution of the captured thermal image.

The predicted thermal image is at a resolution. The resolution of the thermal image is greater than the resolution of the captured thermal image. In some examples, the predicted thermal image is at a voxel-level resolution. An example of voxel-level resolution may be <NUM> x <NUM> pixels. The predicted thermal image or images may be stored in the data store <NUM> as enhanced thermal image data <NUM>. The predicted thermal image or images may be "enhanced" in that the resolution of the predicted thermal image or images is greater than the resolution of the captured thermal image or images. As used herein, the term "enhance" and variations thereof refer to increasing thermal image resolution using a neural network based on a contone map or maps.

Predicting, calculating, or computing the predicted thermal image may include calculating the predicted thermal image of the layer before, at, or after a time that the layer is formed. Accordingly, a thermal image for a layer may be "predicted" before, during, and/or after forming a layer. For example, a thermal image may be predicted for a layer that has not yet been applied and/or printed. Additionally or alternatively, thermal behavior (e.g., a thermal image) may be predicted for a layer at a time after application and/or printing. As used herein, the term "predict" and variants thereof may denote calculation with a machine learning model (e.g., neural network or networks). In some approaches, a predicted thermal image of a layer may be computed independently of capturing a thermal image of the layer.

In some examples, the predicted thermal image may correspond to a layer that is subsequent to a layer corresponding to the captured thermal image. For example, the captured thermal image may correspond to a previous layer k - <NUM> and the predicted thermal image may correspond to a layer k. It should be noted that a number of captured thermal images of previous layers may also be utilized in the calculation in some examples. The contone map or maps may correspond to the same layer (e.g., layer k) as the layer corresponding to the predicted thermal image and/or to a previous layer or layers.

In some examples, the predicted thermal image may correspond to a layer that is the same as a layer corresponding to the captured thermal image. For example, the captured thermal image may correspond to a layer k and the predicted thermal image may correspond to the layer k. It should be noted that a number of captured thermal images of previous layers may also be utilized in the calculation in some examples. The contone map or maps may correspond to the same layer (e.g., layer k) as the layer corresponding to the predicted thermal image and/or to a previous layer or layers.

A contone map may be a representation of agent placement (e.g., placement and/or quantity for a fusing agent and/or placement and/or quantity for a detailing agent). While contone maps are given as examples of data input into the neural network or networks, other information or data may be utilized in addition to contone maps. For example, slices may be utilized to assist predicting thermal images and/or may be utilized as an alternative learning dataset. In particular, slices may be used in addition to a contone map or contone maps in some examples.

It should be noted that other thermal images (e.g., voxel-level captured thermal images) may be utilized to train the neural network or networks in some examples. For instance, the controller <NUM> may compute a loss function based on the predicted thermal image and the thermal image. The neural network or networks may be trained based on the loss function.

In some examples, a neural network may include an input layer or layers, an encoder layer or layers, a spatiotemporal layer (e.g., RNN layer), a decoder layer or layers, and/or an output layer or layers. For example, next to the input layer, an encoder layer may extract features from inputs. The spatiotemporal layer may learn both sequential and spatial information from a contone map or maps and a captured thermal image or images (e.g., from real-time in-machine thermal sensing). The decoder layer may translate features into an output domain and may be situated before the output layer. Each layer may include a node or nodes (e.g., more than one node (or perceptron)) in some implementations. In some examples, a neural network may be connected to another neural network or networks, may include another neural network or networks, and/or may be merged (e.g., stacked) with another neural network or networks. In some examples, another neural network or networks may be utilized as an encoder or decoder. In some examples, multiple encoders or decoders may be utilized, or an encoder or decoder may not be implemented or utilized.

In some examples, the controller <NUM> may upscale the captured thermal image to produce an upscaled thermal image. As used herein, the term "upscaling" and variants thereof denote increasing a resolution of an image. Upscaling may not be based on a contone map and/or may not provide the accuracy of the thermal image enhancement described herein. Examples of upscaling may include interpolation-based approaches, statistical approaches, and/or example-based approaches. For instance, the controller <NUM> may perform bi-cubic interpolation to upscale the captured thermal image to produce the upscaled thermal image.

In some examples, upscaling the captured thermal image may include performing thermal prediction intensity correction as follows. Thermal prediction intensity correction is an empirical approach for thermal image resolution upscaling. This approach may utilize a simple model to upscale a thermal image of a layer (at <NUM> pixels per inch (ppi), for example). Examples of the simple thermal predictive model may include first-principle based models or empirical models. This thermal predictive model upscaling may not utilize a neural network and/or may not utilize a contone map or maps. The thermal image may be down-sampled into a same resolution as low-resolution thermal sensing (e.g., <NUM> x <NUM> pixels). Then, a ratio of measured to predicted temperature may be calculated. For example, an un-distorted infrared camera image (at a resolution of <NUM> x <NUM> pixels, for instance) may be utilized to calculate the ratio of measured to predicted temperatures. The camera image may be utilized to adjust the thermal image that was predicted based on the intensity correction derived from the measured infrared camera image. Interpolation may be utilized to up-sample the calculated ratio to the high resolution (e.g., <NUM> x <NUM> pixels or <NUM> ppi). The high-resolution thermal image may be derived by multiplying the high-resolution ratio by the original thermal image that was predicted.

While the thermal image is upscaled, the generated high-resolution image may show gradients due to interpolation. The enhancement result of the intensity correction may not be accurate enough for some applications. However, this approach may still provide a high-resolution thermal image, which may be utilized to reduce the difficulties in model-based image enhancement. For example, the thermal prediction intensity correction may be utilized in some examples of thermal image enhancement described herein. It should be noted that some examples of thermal image enhancement (e.g., modeling approaches) described herein is not limited to thermal prediction intensity correction. Some examples of thermal image enhancement may utilize any thermal sensing resolution upscaling results as model input. The model may learn how to correct the results during model training.

In some examples, the controller <NUM> may encode, using an encoder (e.g., a first convolutional neural network (CNN)), the upscaled thermal image to produce first data. The first data may include features of the upscaled thermal image. In some examples, the controller <NUM> may encode, using an encoder (e.g., a second convolutional neural network), the fusing contone map, and/or the detailing contone map to produce second data. The second data may include features of the fusing contone map and/or the detailing contone map. In some examples, the controller <NUM> may concatenate the first data with the second data to produce concatenated data. The concatenated data may be input to the neural network (e.g., the recurrent neural network (RNN)). In some examples, the controller <NUM> may decode, using a decoder (e.g., third convolutional neural network), an output of the neural network to produce the predicted thermal image (e.g., the enhanced thermal image).

In some examples, the encoder(s) and/or decoder may be convolutional neural networks, though it should be noted that the encoder(s) and/or decoder may not be convolutional neural networks in some approaches. For example, the encoder(s) and/or decoder may be convolutional neural networks combining different components, including convolutional layers, pooling layers, deconvolutional layers, inception layers, and/or residual layers, etc. The specific architecture should be tuned experimentally.

In some examples, the controller <NUM> may print a layer or layers based on the predicted thermal image. For instance, the controller <NUM> may control the amount and/or location of fusing agent <NUM> and/or detailing agent <NUM> for a layer based on the predicted thermal image. In some examples, the controller <NUM> may drive model setting (e.g., the size of the stride) based on the predicted thermal image (e.g., thermal diffusion). Additionally or alternatively, the controller <NUM> may perform offline print mode tuning based on the predicted thermal image. For example, if the predicted thermal image indicates systematic bias (e.g., a particular portion of the build area is consistently colder or warmer than baseline), the data pipeline may be altered such that the contone maps are modified to compensate for such systematic bias. For instance, if the predicted thermal image indicates a systematic bias, the controller <NUM> may adjust contone map generation (for a layer or layers, for example) to compensate for the bias. Accordingly, the location and/or amount of agent(s) deposited may be adjusted based on the contone map(s) to improve print accuracy and/or performance.

<FIG> is a block diagram illustrating examples of functions that may be implemented to perform thermal mapping. In some examples, one, some, or all of the functions described in connection with <FIG> may be performed by the controller <NUM> described in connection with <FIG>. For instance, instructions for slicing <NUM>, contone map generation <NUM>, neural network or networks <NUM>, and/or operation determination <NUM> may be stored in the data store <NUM> and executed by the controller <NUM> in some examples. In other examples, a function or functions (e.g., slicing <NUM>, contone map generation <NUM>, neural network or networks <NUM>, and/or operation determination <NUM>) may be performed by another apparatus. For instance, slicing <NUM> may be carried out on a separate apparatus and sent to the 3D printing device <NUM>.

3D model data <NUM> may be obtained. For example, the 3D model data <NUM> may be received from another device and/or generated. The 3D model data <NUM> may specify shape and/or size of a 3D model for printing a 3D object. 3D model data <NUM> can define both the internal and the external portion of the 3D object. The 3D model data <NUM> can be defined, for example, using polygon meshes. For example, the 3D model data <NUM> can be defined using a number of formats such as a 3MF file format, an object (OBJ) file format, and/or a stereolithography (STL) file format, among other type of files formats.

Slicing <NUM> may be performed based on the 3D model data <NUM>. For example, slicing <NUM> may include generating a set of 2D slices <NUM> corresponding to the 3D model data <NUM>. In some approaches, the 3D model indicated by the 3D model data <NUM> may be traversed along an axis (e.g., a vertical axis, z-axis, or other axis), where each slice <NUM> represents a 2D cross section of the 3D model. For example, slicing <NUM> the 3D model can include identifying a z-coordinate of a slice plane. The z-coordinate of the slice plane can be used to traverse the 3D model to identify a portion or portions of the 3D model intercepted by the slice plane.

A 3D model and/or stack of 2D slices (e.g., vector slices) may be utilized to generate per-layer machine instructions (e.g., voxel-level agent distribution) by accounting for process physics. Contone maps may be examples of per-layer machine instructions. In some examples, contone map generation <NUM> may be performed based on the slices <NUM>. For example, a contone map or contone maps <NUM> may be generated <NUM> for each slice <NUM>. For instance, contone map generation <NUM> may include generating a fusing contone map and a detailing contone map, where the fusing contone map indicates an area or areas and density distribution for printing fusing agent for a layer. The detailing contone map indicates an area or areas and density distribution for printing detailing agent for the layer. The contone map or maps <NUM> may be represented in a variety of file formats in some examples. For instance, a contone map <NUM> may be formatted as a BKZ contone file, a SIF contone file, and/or another kind of contone file.

The neural network or networks <NUM> may be used to calculate (e.g., predict) a predicted thermal image <NUM> (e.g., enhanced thermal image) based on the contone map or maps <NUM> and thermal image data <NUM> from the same layer and/or a previous layer or layers. The thermal image data <NUM> may represent a thermal image or images at a first resolution (e.g., <NUM> x <NUM> pixels or <NUM> x <NUM> pixels). The predicted thermal image <NUM> may be at a second resolution (e.g., <NUM> x <NUM> pixels).

Operation determination <NUM> may be performed based on the predicted thermal image <NUM>. For example, the operation determination <NUM> may produce an operation signal <NUM> indicating control information. The control information may be utilized to print a layer or layers based on the predicted thermal image <NUM>. For instance, the operation signal <NUM> may indicate controlling the amount and/or location of fusing agent and/or detailing agent for a layer based on the predicted thermal image <NUM>. In some examples, the operation signal <NUM> may drive model setting (e.g., the size of the stride) based on the predicted thermal image <NUM> (e.g., thermal diffusion). Additionally or alternatively, the operation signal <NUM> may indicate offline print mode tuning based on the predicted thermal image <NUM>. For example, if the predicted thermal image <NUM> indicates systematic bias (e.g., a particular portion of the build area is consistently colder or warmer than baseline), the data pipeline may be altered such that the contone maps are modified to compensate for such systematic bias. For instance, if the predicted thermal image <NUM> indicates a systematic bias, the operation signal <NUM> may indicate an adjustment to contone map generation (for a layer or layers, for example) to compensate for the bias. Accordingly, the location and/or amount of agent(s) deposited may be adjusted based on the contone map(s) to improve print accuracy and/or performance. In some examples, performing an operation may include presenting the thermal image(s) (e.g., predicted thermal image) on a display and/or sending the thermal image(s) (e.g., predicted thermal image(s)) to another device.

<FIG> is a block diagram of an example of an apparatus <NUM> that may be used in thermal mapping. The apparatus <NUM> may be a computing device, such as a personal computer, a server computer, a printer, a 3D printer, a smartphone, a tablet computer, etc. The apparatus <NUM> may include and/or may be coupled to a processor <NUM>, a data store <NUM>, an input/output interface <NUM>, a machine-readable storage medium <NUM>, and/or a thermal image sensor or sensors <NUM>. In some examples, the apparatus <NUM> may be in communication with (e.g., coupled to, have a communication link with) an additive manufacturing device (e.g., the 3D printing device <NUM> described in connection with <FIG>). Alternatively, the apparatus <NUM> may be an example of the 3D printing device <NUM> described in connection with <FIG>. For instance, the processor <NUM> may be an example of the controller <NUM> described in connection with <FIG>, the data store <NUM> may be an example of the data store <NUM> described in connection with <FIG>, and the thermal image sensor or sensors <NUM> may be an example of the thermal sensor <NUM> described in connection with <FIG>. The apparatus <NUM> may include additional components (not shown) and/or some of the components described herein may be removed and/or modified without departing from the scope of this disclosure.

The processor <NUM> may be any of a central processing unit (CPU), a semiconductor-based microprocessor, graphics processing unit (GPU), FPGA, an application-specific integrated circuit (ASIC), and/or other hardware device suitable for retrieval and execution of instructions stored in the machine-readable storage medium <NUM>. The processor <NUM> may fetch, decode, and/or execute instructions (e.g., operation instructions <NUM>) stored on the machine-readable storage medium <NUM>. Additionally or alternatively, the processor <NUM> may include an electronic circuit or circuits that include electronic components for performing a functionality or functionalities of the instructions (e.g., operation instructions <NUM>). In some examples, the processor <NUM> may be configured to perform one, some, or all of the functions, operations, steps, methods, etc., described in connection with one, some, or all of <FIG> and/or <NUM>-<NUM>.

The machine-readable storage medium <NUM> may be any electronic, magnetic, optical, or other physical storage device that contains or stores electronic information (e.g., instructions and/or data). Thus, the machine-readable storage medium <NUM> may be, for example, Random Access Memory (RAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), a storage device, an optical disc, and the like. In some implementations, the machine-readable storage medium <NUM> may be a non-transitory tangible machine-readable storage medium, where the term "non-transitory" does not encompass transitory propagating signals.

The apparatus <NUM> may also include a data store <NUM> on which the processor <NUM> may store information. The data store <NUM> may be volatile and/or non-volatile memory, such as Dynamic Random Access Memory (DRAM), EEPROM, magnetoresistive random-access memory (MRAM), phase change RAM (PCRAM), memristor, flash memory, and the like. In some examples, the machine-readable storage medium <NUM> may be included in the data store <NUM>. Alternatively, the machine-readable storage medium <NUM> may be separate from the data store <NUM>. In some approaches, the data store <NUM> may store similar instructions and/or data as that stored by the machine-readable storage medium <NUM>. For example, the data store <NUM> may be non-volatile memory and the machine-readable storage medium <NUM> may be volatile memory.

The apparatus <NUM> may further include an input/output interface <NUM> through which the processor <NUM> may communicate with an external device or devices (not shown), for instance, to receive and store the information pertaining to the object or objects to be manufactured (e.g., printed). The input/output interface <NUM> may include hardware and/or machine-readable instructions to enable the processor <NUM> to communicate with the external device or devices. The input/output interface <NUM> may enable a wired or wireless connection to the external device or devices. The input/output interface <NUM> may further include a network interface card and/or may also include hardware and/or machine-readable instructions to enable the processor <NUM> to communicate with various input and/or output devices, such as a keyboard, a mouse, a display, another apparatus, electronic device, computing device, etc., through which a user may input instructions into the apparatus <NUM>.

In some examples, the machine-readable storage medium <NUM> may store thermal image data <NUM>. The thermal image data <NUM> may be obtained (e.g., received) from a thermal image sensor or sensors <NUM> and/or may be predicted. For example, the processor <NUM> may execute instructions (not shown in <FIG>) to obtain a captured thermal image or images for a layer or layers. In some examples, the apparatus <NUM> may include a thermal image sensor or sensors <NUM>, may be coupled to a remote thermal image sensor or sensors, and/or may receive thermal image data <NUM> (e.g., a thermal image or images) from a (integrated and/or remote) thermal image sensor. Some examples of thermal image sensors <NUM> include thermal cameras (e.g., infrared cameras). Other kinds of thermal sensors may be utilized. In some examples, thermal sensor resolution may be less than voxel resolution (e.g., each temperature readout may cover an area that includes multiple voxels). For example, a low-resolution thermal camera with a low-resolution (e.g., <NUM> x <NUM> pixels, <NUM> x <NUM> pixels, etc.) may be utilized. In other examples, a high-resolution thermal image sensor or sensors <NUM> may provide voxel-level (or near voxel-level) thermal sensing (e.g., <NUM> x <NUM> pixels) for neural network training.

The thermal image data <NUM> may include a thermal image or images. As described above, a thermal image may be an image that indicates heat (e.g., temperature) over an area and/or volume. For example, a thermal image may indicate a build area temperature distribution (e.g., thermal temperature distribution over a top layer). In some examples, the thermal image sensor or sensors <NUM> may undergo a calibration procedure to overcome distortion introduced by the thermal image sensor or sensors <NUM>. For example, a thermal image may be transformed to register the thermal image with the contone map or maps. Different types of thermal sensing devices may be used in different examples.

In some examples, the processor <NUM> may execute contone map obtaining instructions <NUM> to obtain contone map data <NUM>. For example, the contone map obtaining instructions <NUM> may generate a contone map or maps (e.g., from slice data and/or 3D model data) and/or may receive a contone map or maps from another device (via the input/output interface <NUM>, for example). The contone map data <NUM> may indicate agent distribution (e.g., fusing agent distribution and/or detailing agent distribution) at the voxel level for printing a 3D object. For instance, the contone map data <NUM> may be utilized as per-layer machine instructions (e.g., voxel-level machine instructions) for agent distribution.

It should be noted that multiple different agent contone maps corresponding to different abilities to absorb or remove thermal energies may be utilized in some examples. Additionally or alternatively, some examples may utilize different print modes where multiple contone maps may be used for each agent.

For a given layer (e.g., a current layer, a top layer, etc.), the contone map or maps of all agents deposited to the layer may be an energy driving force in some examples. It should be noted that another voxel-level energy influencer may include neighboring voxels in previous layers that may have a temperature differential compared to a given voxel, which may induce heat flux into or out of the voxel.

The machine-readable storage medium <NUM> may store neural network data <NUM>. The neural network data <NUM> may include data defining and/or implementing a neural network or neural networks. For instance, the neural network data <NUM> may define a node or nodes, a connection or connections between nodes, a network layer or network layers, and/or a neural network or neural networks. In some examples, the processor <NUM> may utilize (e.g., execute instructions included in) the neural network data <NUM> to calculate predicted thermal images. A predicted thermal image or images may be stored as enhanced thermal image data <NUM> on the machine-readable storage medium <NUM>.

In some examples, the processor <NUM> uses the neural network or networks (defined by the neural network data <NUM>) to enhance the captured thermal image or images. For example, the processor <NUM> may enhance the captured thermal image using a neural network or networks based on the contone map or maps to produce an enhanced thermal image or images. The enhanced thermal image(s) may have an increased resolution relative to a resolution of the captured thermal image(s). The enhanced thermal image or images may be stored as enhanced thermal image data <NUM>. For instance, the processor <NUM> may calculate (e.g., predict), using a neural network or a plurality of neural networks, a predicted thermal image of a layer based on a captured thermal image or a plurality of captured thermal images and a contone map or a plurality of contone maps (e.g., a fusing contone map and a detailing contone map).

Enhancing a captured thermal image (e.g., predicting, calculating, or computing the predicted thermal image) may include calculating the enhanced thermal image of the layer before, at, or after a time that the layer is formed. In some examples, the enhanced thermal image may correspond to a layer that is subsequent to a layer corresponding to the captured thermal image. For example, the captured thermal image may correspond to a previous layer k - <NUM> and the enhanced thermal image may correspond to a layer k. It should be noted that a number of captured thermal images of previous layers may also be utilized in the calculation in some examples. The contone map or maps may correspond to the same layer (e.g., layer k) as the layer corresponding to the enhanced thermal image and/or to a previous layer or layers.

In some examples, the enhanced thermal image may correspond to a layer that is the same as a layer corresponding to the captured thermal image. For example, the captured thermal image may correspond to a layer k and the enhanced thermal image may correspond to the layer k. It should be noted that a number of captured thermal images of previous layers may also be utilized in the calculation in some examples. The contone map or maps may correspond to the same layer (e.g., layer k) as the layer corresponding to the enhanced thermal image and/or to a previous layer or layers.

In some examples, the processor <NUM> may execute the operation instructions <NUM> to perform an operation based on the enhanced thermal image. For example, the processor <NUM> may print (e.g., control amount and/or location of agent(s) for) a layer or layers based on the predicted thermal image <NUM>. In some examples, the processor <NUM> may drive model setting (e.g., the size of the stride) based on the enhanced thermal image. Additionally or alternatively, the processor <NUM> may perform offline print mode tuning based on the enhanced thermal image. Additionally or alternatively, the processor <NUM> may send a message (e.g., alert, alarm, progress report, quality rating, etc.) based on the enhanced thermal image. Additionally or alternatively, the processor <NUM> may halt printing in a case that the enhanced thermal image indicates a problem (e.g., more than a threshold difference between a layer or layers of printing and the 3D model and/or slices). Additionally or alternatively, the processor <NUM> may feed the predicted thermal image for the upcoming layer to a thermal feedback control system to online compensate the contone maps for the upcoming layer.

Examples of the techniques described herein may utilize a deep neural network based practical model training approach. The approach can achieve voxel level thermal prediction with built-in low-resolution thermal sensing and contone maps as input. The approach can achieve the prediction that is approximately <NUM> times the resolution of built-in thermal sensing. This approach may enable real-time in-situ voxel-level thermal image prediction and feedback control. For example, the neural network architecture may enable the real-time in-situ fusing layer thermal prediction with print resolution and/or online closed-loop thermal feedback control. Some examples of the techniques described here may enable additive manufacturing devices to provide built-in online voxel-level high-resolution thermal sensing.

Some examples of the techniques described herein may utilize a neural network architecture based approach that accounts for different thermal drivers to predict the voxel level high-resolution thermal behavior from low-resolution thermal sensing and voxel-level agent contone maps. Some examples may infer the missing details from additional information and achieve high-resolution thermal prediction.

Some examples of the techniques described herein may provide voxel-level fusing layer thermal prediction (e.g., future thermal image prediction) using built-in low-resolution thermal sensors. In some examples, performing the operation may include using the thermal prediction to serve as an online thermal prediction engine to enable voxel-level thermal feedback control. For example, the processor <NUM> may perform feedback control by controlling the printing process and/or the machine instructions. Additionally or alternatively, processor <NUM> may utilize the thermal prediction to serve as a prediction engine (e.g., online and/or offline) for a variety of 3D printer based analysis, monitoring, diagnosis, and/or control, etc. Additionally or alternatively, processor <NUM> may utilize the thermal prediction to serve as an offline simulation tool for thermal behavior prediction, visualization, and/or quantification, etc..

In some examples, a neural network may be utilized to predict a thermal image of the same layer as the layer of the captured thermal image. For example, a low-resolution captured thermal image may be utilized to produce an enhanced voxel-level resolution thermal image. Some examples may be utilized online and/or offline to provide voxel-level thermal image sensing. The voxel-level (e.g., high-accuracy) thermal sensing may be used in modeling, machine behavior analysis, and/or thermal analysis, etc..

In some examples, the machine-readable storage medium <NUM> may store 3D model data (not shown in <FIG>). The 3D model data may be generated by the apparatus <NUM> and/or received from another device. In some examples, the machine-readable storage medium <NUM> may include slicing instructions (not shown in <FIG>). For example, the processor <NUM> may execute the slicing instructions to perform slicing on the 3D model data to produce a stack of 2D vector slices.

In some examples, the operation instructions <NUM> may include 3D printing instructions. For instance, the processor <NUM> may execute the 3D printing instructions to print a 3D object or objects. In some implementations, the 3D printing instructions may include instructions for controlling a device or devices (e.g., rollers, print heads, and/or thermal projectors, etc.). For example, the 3D printing instructions may use a contone map or contone maps (stored as contone map data, for instance) to control a print head or heads to print an agent or agents in a location or locations specified by the contone map or maps. In some examples, the processor <NUM> may execute the 3D printing instructions to print a layer or layers. The printing (e.g., thermal projector control) may be based on thermal images (e.g., captured thermal images and/or predicted thermal images).

In some examples, the machine-readable storage medium <NUM> may store neural network training instructions. The processor <NUM> may execute the neural network training instructions to train a neural network or neural networks (defined by the neural network data <NUM>, for instance). In some examples, the processor <NUM> may train the neural network or networks using a set of training thermal images. The set of training thermal images may have a resolution that is greater than the resolution of a captured thermal image (e.g., anticipated captured thermal image at run-time). For example, a training thermal sensor may have a voxel-level resolution for training. The training thermal sensor may capture the set of training thermal images. In some examples, the training thermal sensor may be placed outside of an additive manufacturing device (e.g., printer). In some approaches, the neural network training instructions may include a loss function. The processor <NUM> may compute the loss function based on a predicted thermal image and a training thermal image. For example, the training thermal image may provide the ground truth (which may or may not be at voxel-level) for the loss function. The loss function may be utilized to train a neural network or neural networks. For example, a node or nodes and/or a connection weight or weights in the neural network or networks may be adjusted based on the loss function in order to improve the prediction accuracy of the neural network or networks. It should be noted that not all of the steps and/or features described in connection with <FIG> may be required in all implementations.

<FIG> is a flow diagram illustrating an example of a method <NUM> for thermal mapping. The method <NUM> and/or a method <NUM> step or steps may be performed by an electronic device. For example, the method <NUM> may be performed by the apparatus <NUM> described in connection with <FIG> (and/or by the 3D printing device <NUM> described in connection with <FIG>).

The apparatus <NUM> obtains <NUM> a map. A map is a set of image data for additive manufacturing. Examples of maps include shape maps, slice data, contone maps, etc. A shape map may indicate a shape or shapes (e.g., geometrical data) for additive manufacturing. For example, the apparatus <NUM> may generate a contone map or maps (e.g., fusing agent contone map and/or detailing agent contone map) based on 3D model data and/or slice data. The contone map is at a voxel-level resolution. It should be note that a map (e.g., a shape map, slice data, etc.) at a voxel-level resolution may be utilized in addition to the contone map(s) described herein to calculate a predicted thermal image.

The apparatus <NUM> may obtain <NUM>, for the layer, a first thermal image. For example, after the layer has been deposited, the apparatus <NUM> may obtain <NUM> a captured thermal image of the layer using a thermal image sensor or may receive a captured thermal image of the layer from a remote image sensor. The first thermal image (e.g., captured thermal image) may be at a low resolution.

The apparatus <NUM> determines <NUM>, using a neural network, a second thermal image at a second resolution based on the contone map and the first thermal image. The second resolution is greater than the first resolution. The contone map is at a voxel-level resolution, and the second resolution of the second thermal image is at the voxel-level resolution. The second resolution (e.g., <NUM> x <NUM> pixels) may be greater than the first resolution (e.g., <NUM> x <NUM> pixels, <NUM> x <NUM> pixels, etc.) by a factor of at <NUM> or more.

In some examples, the first thermal image corresponds to a first layer (e.g., k) and the second thermal image corresponds to the first layer (e.g., k). In some examples, the first thermal image corresponds to a first layer (e.g., k - <NUM>) and the second thermal image corresponds to a second layer (e.g., k) that is subsequent to the first layer.

In some examples, at least one neural network may utilize the contone map or maps (e.g., voxel-level machine instructions) and/or captured thermal image or images to calculate an enhanced thermal image. In some examples, the neural network is a recurrent neural network including one or multiple stacked convolutional long short-term memory networks.

In some examples, the apparatus <NUM> may increase the first resolution of the first thermal image to produce an upscaled first thermal image. For example, the apparatus <NUM> may upscale the first thermal image (e.g., low-resolution captured thermal image) using an empirical model, an interpolation-based approach, statistical approach, example-based approach, and/or thermal prediction intensity correction.

In some examples, the apparatus <NUM> may encode the upscaled first thermal image to produce first data that is provided to the neural network. In some examples, encoding the first thermal image is performed with a first neural network (e.g., a convolutional neural network). The apparatus <NUM> may encode the contone map to produce second data that is provided to the neural network. In some examples, encoding the contone map is performed with a second neural network (e.g., a convolutional neural network). The apparatus <NUM> may decode an output of the neural network to produce the second thermal image (e.g., enhanced thermal image). In some examples, decoding the output is performed with a third neural network.

In some examples, the apparatus <NUM> may concatenate the first data and the second data, where the first data is based on the first thermal image and the second data corresponds to the contone map. Concatenating the first and second data may include combining the feature layers. In an example, the dimensionality of the first data is <NUM>*<NUM>*<NUM> and the dimensionality of the second data is <NUM>*<NUM>*<NUM>, where <NUM>*<NUM> is the image size and <NUM> is the number of feature layers. In this example, the concatenated data has a dimensionality of <NUM>*<NUM>*<NUM>.

<FIG> is a diagram illustrating an example of a neural network architecture <NUM>. The neural network architecture <NUM> described in connection with <FIG> may be an example of the neural networks described in connection with <FIG>. The neural network architecture <NUM> may take into account voxel-level thermal influencers to the fusing layer. A deep neural network with the neural network architecture <NUM> may learn spatiotemporal information, in recognition of two thermal influencers to the fusing layer thermal behavior: the energy absorption and/or loss driven by contone maps 586a-n, and the voxel-level thermal coupling both within a layer and among different layers. The network architecture <NUM> may include a spatiotemporal neural network <NUM>. An example of a spatiotemporal neural network <NUM> is a recurrent neural network. In some examples, the spatiotemporal neural network <NUM> may include one or multiple (e.g., two or more) stacked Convolutional Long Short-Term Memory networks (Conv-LSTM). A Conv-LSTM is a type of recurrent neural network that overcomes numerical instability issues and takes spatial and temporal influence into account.

At each layer, the current layer contone maps (or data based on the contone maps) and a previous layer thermal image (or data based on the previous layer thermal image) may be utilized as input. For example, second data based on k - nth layer contone maps 586n may be concatenated <NUM> with first data based on a (k - n - <NUM>)-th layer upscaled thermal image 590n. Similarly, second data based on k-th layer contone maps 586a may be concatenated with first data based on a (k -<NUM>)-th layer upscaled thermal image 590a. Accordingly, the spatiotemporal neural network <NUM> may learn from the historical sequential information from both contone maps 586a-n and thermal images 590a-n.

The sequence of thermal images 590a-n may provide information regarding inter-layer thermal diffusion. The sequence of contone maps 586a-n may reflect energy application for each layer, and may also provide the material and/or phase information of previous layers. Accordingly, the sequence of contone maps 586a-n may help the spatiotemporal neural network <NUM> to learn heat flux behavior. Additionally, the sequence of contone maps 586a-n may provide voxel-level detail information, which may enable the spatiotemporal neural network <NUM> to infer an increased resolution (e.g., voxel-level) thermal image from the low-resolution thermal image.

In some examples, the following steps may be performed to calculate a predicted thermal image <NUM> of the k-th layer (e.g., the fusing layer): the k-th layer contone maps 586a may be passed through a contone encoder <NUM> and an upscaled thermal image 590a of the previous (k-<NUM>)-th layer (e.g., last buried layer) may be passed through a thermal image encoder <NUM> separately. It should be noted that the previous layer upscaled thermal image 590a may be based on a previous layer captured low-resolution thermal image. The thermal image encoder <NUM> may encode the (k - <NUM>)-th layer upscaled thermal image 590a to produce first data (e.g., features). The contone encoder <NUM> may encode the k-th layer contone maps 586a to produce second data (e.g., features).

The first data (e.g., features) and the second data (e.g., features) may be concatenated as the input to the spatiotemporal (e.g., Conv-LSTM) neural network <NUM>. The output <NUM> for the k-th layer (at the current timestamp, for example) may be passed through a decoder <NUM> to produce the predicted thermal image <NUM> for the k-th layer (e.g., the fusing layer).

In some examples, the thermal image encoder <NUM> is a CNN, the contone encoder <NUM> is a CNN, and the decoder <NUM> is a CNN. For instance, the thermal image encoder <NUM>, the contone encoder <NUM>, and/or the decoder <NUM> may be a variety of CNNs combining different components, e.g., convolutional layers, pooling layers, de-convolutional layers, inception layers, residual layers, etc. The architectures of the thermal image encoder <NUM>, the contone encoder <NUM>, and/or the decoder <NUM> may be tuned experimentally. In some examples, inception module based CNNs may be utilized for the contone encoder <NUM>, the thermal image encoder <NUM>, and/or the decoder <NUM>.

In some examples, built-in low-resolution thermal sensing may provide low-resolution thermal images. In some examples, the low-resolution thermal images may not be directly utilized. For example, upscaling (e.g., thermal prediction intensity correction) may be performed on the low-resolution thermal images to generate upscaled thermal images 590a-n. The upscaled thermal images 590a-n may be provided to the thermal image encoder <NUM>. The features from the thermal image encoder <NUM> may be provided to the spatiotemporal neural network <NUM> as input.

The neural network architecture <NUM> may learn additional information from the sequence of contone maps 586a-n, and learn how to correct the current upscaled (e.g., voxel-level) thermal image 590a to produce a more accurate predicted (e.g., enhance, voxel-level) thermal image <NUM>.

The neural network architecture <NUM> may achieve thermal image resolution enhancement and fusing layer thermal prediction concurrently because the neural network architecture <NUM> may be designed to learn the information required for both. Accordingly, some examples of the techniques and the neural network architecture <NUM> described, may avoid additional processing for increasing thermal image resolution while achieving enhanced thermal image prediction.

<FIG> is a diagram illustrating another example of a neural network architecture <NUM>. The neural network architecture <NUM> described in connection with <FIG> may be an example of the neural networks described in connection with <FIG>. In this example, a low-resolution (e.g., <NUM> x <NUM> pixels, etc.) thermal image may be enhanced to a high-resolution (e.g., voxel-level, <NUM> x <NUM> pixels, etc.) thermal image. The image is enhanced by a factor greater than or equal to <NUM>, approximately <NUM>, etc.. In some examples, <NUM> pixels may be predicted from one low-resolution image pixel, which is difficult to achieve with accuracy in other approaches. In some of the techniques described herein, additional information may be utilized to infer the <NUM> pixels per one low-resolution pixel, for example.

In some examples, the additional information may include a sequence of contone maps 686a-n (e.g., fusing agent contone map and/or detailing agent contone map), and a sequence of captured thermal images 690a-n. For example, two thermal influencers to the fusing layer may include a sequence of previous thermal images, which drive heat transfer, and a sequence of fusing agent and detailing agent contone maps, which drive the layer energy application. This information may help to infer the thermal behavior of the current layer. In some examples, sequence of contone maps 686a-n may be high-resolution (e.g., voxel-level) images and may reflect the material phases, which may provide the voxel-level information to infer the thermal voxels. Accordingly, some of the techniques described herein may be based on a deep neural network for voxel-level thermal prediction with built-in low-resolution thermal sensing, to produce enhance resolution thermal images.

The neural network architecture <NUM> may be utilized to predate a high-resolution (e.g., voxel-level) thermal image of a current layer k. At each layer, the layer contone maps (or data based on the contone maps) and a layer thermal image (or data based on the current layer thermal image) may be utilized as input. For example, second data based on (k - n)-th layer contone maps 686n may be concatenated <NUM> with first data based on a (k - n)-th layer upscaled thermal image 690n. Similarly, second data based on k-th layer contone maps 686a may be concatenated with first data based on a k-th layer upscaled thermal image 690a. Accordingly, the spatiotemporal neural network <NUM> may learn from the historical sequential information from both contone maps 686a-n and thermal images 690a-n.

In some examples, the following steps may be performed to calculate a predicted thermal image <NUM> of the k-th layer (e.g., the fusing layer): the k-th layer contone maps 686a may be passed through a contone encoder <NUM> and an upscaled thermal image 690a of the k-th layer may be passed through a thermal image encoder <NUM> separately. It should be noted that this aspect of this example may differ from the example described in connection with <FIG>.

It should be noted that the upscaled thermal image 690a may be based on a captured low-resolution thermal image. The thermal image encoder <NUM> may encode the k-th layer upscaled thermal image 690a to produce first data (e.g., features). The contone encoder <NUM> may encode the k-th layer contone maps 686a to produce second data (e.g., features).

The first data (e.g., features) and the second data (e.g., features) may be concatenated as the input to the spatiotemporal (e.g., Conv-LSTM) neural network <NUM>. The output <NUM> for the k-th layer (at the current timestamp, for example) may be passed through a decoder <NUM> to produce the predicted thermal image <NUM> for the k-th layer.

The network architecture <NUM> may include a spatiotemporal neural network <NUM>. An example of a spatiotemporal neural network <NUM> is a recurrent neural network. In some examples, the spatiotemporal neural network <NUM> may include one or multiple stacked Conv-LSTM networks. A Conv-LSTM is a type of recurrent neural network that overcomes numerical instability issues.

It should be noted that while the neural network architecture <NUM> of <FIG> has some similarity with the neural network architecture <NUM> of <FIG>, the detailed architecture, including the spatiotemporal neural network <NUM> (e.g., Conv-LSTM), the contone encoder <NUM>, and the decoder <NUM>, are not necessarily the same architecture as that described in connection with <FIG>. The input components may play different roles in the modeling due to different objectives. In some approaches, the specific architecture may be tuned experimentally. In some examples, the thermal image encoder <NUM> is a CNN, the contone encoder <NUM> is a CNN, and the decoder <NUM> is a CNN. For instance, the thermal image encoder <NUM>, the contone encoder <NUM>, and/or the decoder <NUM> may be a variety of CNNs combining different components, e.g., convolutional layers, pooling layers, de-convolutional layers, inception layers, residual layers, etc. The architectures of the thermal image encoder <NUM>, the contone encoder <NUM>, and/or the decoder <NUM> may be tuned experimentally. In some examples, inception module based CNNs may be utilized for the contone encoder <NUM>, the thermal image encoder <NUM>, and/or the decoder <NUM>.

In some examples, low-resolution thermal sensing may provide low-resolution thermal images. In some examples, the low-resolution thermal images may not be directly utilized. For example, upscaling (e.g., thermal prediction intensity correction) may be performed on the low-resolution thermal images to generate upscaled thermal images 690a-n. The upscaled thermal images 690a-n may be provided to the thermal image encoder <NUM>. The features from the thermal image encoder <NUM> may be provided to the spatiotemporal neural network <NUM> as input.

Providing an upscaled (e.g., voxel-level) thermal image instead of the original low-resolution thermal image may be a beneficial approach to synthesize the additional information and improve inference of the enhanced thermal voxels. For example, the thermal prediction intensity correction may utilize a simple thermal prediction for resolution enhancement, which may provide detailed information. While the upscaled thermal image itself may not provide enough accuracy at the voxel-level in some examples, the upscaled thermal image may achieve some accuracy at a resolution level higher than the low-resolution thermal sensing, making resolution enhancement easier. This may be beneficial since deep neural networks that could directly enhance the thermal images by a factor of <NUM> (e.g., a de-convolutional NN) may require a large number of parameters, which may make model training and tuning extremely difficult. Instead, using the upscaled thermal image and training a model to correct the voxel level value may require significantly fewer parameters and a simpler network. Accordingly, some examples of the neural network architecture <NUM> may be beneficial to enhance thermal image resolution. For example, the neural network architecture <NUM> may make use of physical behavior information may be designed for thermal images specifically.

Due to the additional information extraction and inference from the sequence of contone maps 686a-n and upscaling (e.g., thermal prediction intensity correction), the neural network architecture <NUM> may achieve thermal image resolution enhancement by a factor of <NUM>. Accordingly, some examples of the techniques described herein may achieve a degree of image resolution enhancement that is much greater than that of other approaches for increasing image resolution.

<FIG> is a block diagram illustrating examples of neural network training <NUM> and prediction <NUM>. Some examples of the neural networks described herein may be trained and/or may perform prediction in accordance with the examples described in connection with <FIG>.

Training <NUM> may include obtaining low-resolution raw thermal image(s) 705a, high-resolution (e.g., voxel-level) raw training thermal image(s) <NUM>, and contone maps 717a (e.g., voxel-level contone maps) from a printer data pipeline. Undistortion 707a may be applied to the raw thermal image(s) 705a to produce undistorted thermal image(s) 709a. For example, the raw thermal images 705a may be captured with a fisheye lens, and the undistortion 707a may remove distortion resulting from the fisheye lens. Upscaling 711a (e.g., thermal prediction intensity correction) may be applied to the undistorted thermal image(s) 709a to produce upscaled (e.g., voxel-level) thermal image(s) 715a. Keystone correction <NUM> may be applied to the raw training thermal image(s) <NUM> to produce undistorted training thermal image(s) <NUM> (at the voxel level, for example).

The upscaled thermal image 715a and the contone maps 717a (e.g., fusing agent contone map and/or detailing agent contone map) may be fed as input into the neural network with alignment, cropping, and sequence generation 725a for model training <NUM>. The undistorted training thermal image(s) <NUM> may be utilized as ground truth data for model training <NUM>.

A similar procedure may be utilized in prediction <NUM> (e.g., while online, during run-time, etc.). For example, low-resolution raw thermal image(s) 705b and contone maps 717b (e.g., voxel-level contone maps) may be obtained from a printer data pipeline. Undistortion 707b may be applied to the raw thermal image(s) 705b to produce undistorted thermal image(s) 709a. Upscaling 711b (e.g., thermal prediction intensity correction) may be applied to the undistorted thermal image(s) 709b to produce upscaled (e.g., voxel-level) thermal image(s) 715b.

The upscaled thermal image 715b and the contone maps 717b (e.g., fusing agent contone map and/or detailing agent contone map) may be fed as input into the neural network with sequence generation 725b to produce predicted (e.g., enhanced) thermal image(s) <NUM>. For example, the trained neural network may calculate the high-resolution (e.g., voxel-level) predicted thermal image(s) <NUM>. No training thermal image may be utilized in prediction <NUM>.

<FIG> includes images illustrating an example of thermal mapping. Each of the images corresponds to a single layer. The top left image is an example of a fusing contone map <NUM>. The top middle image is an example of a detailing contone map <NUM>. The top right image <NUM> is a high-resolution (e.g., voxel-level) thermal image (e.g., ground truth). The bottom left image is an example of an approach for simple thermal prediction <NUM> using an empirical model. The bottom middle image is an upscaled thermal image <NUM> generated from thermal intensity correction. The bottom right image is a predicted image <NUM> in accordance with some of the techniques disclosed herein.

This example demonstrates that some examples of the neural network architecture described herein may effectively achieve the voxel-level fusing layer (e.g., upcoming layer) thermal prediction that is approximately <NUM> times the thermal sensing resolution. The neural network architecture may effectively capture the thermal behavior of energy application driven by contone maps and inter-layer thermal diffusion. This example also illustrates that some of the techniques described herein may overcome a blurring effects introduced by thermal intensity correction, which demonstrates improved accuracy.

<FIG> includes images illustrating another example of thermal mapping. Each of the images corresponds to a single layer. The top left image is an example of a fusing contone map <NUM>. The top middle image is an example of a detailing contone map <NUM>. The top right image <NUM> is a high-resolution (e.g., voxel-level) thermal image (e.g., ground truth). The bottom left image is a low-resolution captured thermal image <NUM>. The bottom middle image is an upscaled thermal image <NUM> generated from thermal intensity correction. The bottom right image is a predicted image <NUM> in accordance with some of the techniques disclosed herein. This example demonstrates that some examples of the neural network architecture described herein may effectively achieve the voxel-level thermal resolution enhancement that is approximately <NUM> times the thermal sensing resolution.

Claim 1:
A computer-implemented method (<NUM>) for thermal mapping for additive manufacturing by an electronic device, comprising:
obtaining (<NUM>) a two dimensional contone map, at a thermal voxel level resolution, for additive manufacturing, wherein the contone map is a representation of agent placement during additive manufacturing;
obtaining (<NUM>) a first thermal image at a first resolution, whereby the first resolution is a captured thermal image resolution; and
calculating (<NUM>), using a neural network, a second thermal image at a second resolution based on the map and the first thermal image, wherein the second resolution is at the voxel-level resolution and greater than the first resolution by a factor of at least <NUM>.