Fully in tensor computation manner: one‐shot dense 3D structured light and beyond

Cognitive Computation and SystemsVolume 2, Issue 4 p. 262-272 Research ArticleOpen Access Fully in tensor computation manner: one-shot dense 3D structured light and beyond Xuan-Li Chen, Corresponding Author Xuan-Li Chen xuanli.chen@icloud.com orcid.org/0000-0002-3058-2262 PSI-ESAT, KU Leuven, BelgiumSearch for more papers by this authorLuc Van Gool, Luc Van Gool PSI-ESAT, KU Leuven, Belgium CVLab, ETH Zurich, SwitzerlandSearch for more papers by this author Xuan-Li Chen, Corresponding Author Xuan-Li Chen xuanli.chen@icloud.com orcid.org/0000-0002-3058-2262 PSI-ESAT, KU Leuven, BelgiumSearch for more papers by this authorLuc Van Gool, Luc Van Gool PSI-ESAT, KU Leuven, Belgium CVLab, ETH Zurich, SwitzerlandSearch for more papers by this author First published: 03 December 2020 https://doi.org/10.1049/ccs.2019.0027AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Tensor computation evolves fast towards a prosperous existence in recent years, e.g. PyTorch. An immediate advantage of using tensor computation is that one does not need to implement low-level parallelism to attain efficient computation, which is of simplicity for both research and application development. The authors began with discovering that a simple manoeuvre 'tensor shift' could perform neighbourhood manipulation in very efficient parallel manner. Based on 'tensor shift', they derive the tensor version of a renowned correspondence search algorithm: semi-global matching (SGM), which they prefix the name as tensor-SGM. To evaluate their idea, they build-up a novel and practical one-shot structured light 3D acquisition system, which yields state-of-art reconstruction results using off-the-shelf hardware. This is the first fully tensorised 3D reconstruction system published to the authors' best knowledge, and it opens new possibilities. A major one is, in the same tensorised framework, they solved the pattern interfering problem which hinders multi-structured light systems from working together. This part is marked as 'beyond' in this study to avoid confusing the readers the spotlight: the fully tensorised 3D structured light framework. 1 Introduction Computation efficiency is now-a-days fundamental requirement, and several parallel computing development kits are released (e.g. different GPUs/TPUs/'AI Chips' with their tool-kits). As tools, not only the running performance but also the corresponding learning curves matter. Basically, one has to spend weeks to learn one parallel pipeline and months before venturing to productive development for either research or industry (e.g. smart phone apps). Few people could afford to devote such long time to build-up this skill, and even the skill itself is rather spontaneous as we are in a time in which computation platforms seem to evolve. Industry is making tremendous endeavour to provide better computing products. Tensor computation platforms (e.g. PyTorch [1], Tensorflow) come into existence and provide a high level API to access this effective computation power, which renders possible an efficient access to tame parallel computing. A complete course (e.g. tutorial from corresponding site) allows one to get started in a rather short time (a few days, depends on one's previous programming experience). In other words, it increases user friendliness for rigorous algorithms. On the other hand, extensible capability is attained as these platforms seem to bridge also hardware gaps during development (e.g. clouds, clusters, desktop GPUs and mobile devices). In short, companies (e.g. Nvidia, Cambricon [2] etc.) release evolving hardware to support tensor computation platforms to keep improving the computational performance. Computer vision, as an engineering science, is influenced not only by the correctness of the theory but also the computing to approximate the solution. The first part of our paper is based on semi-global matching (SGM) [3], which is an good example. SGM makes significant contributions to both academy and industry by making dense corresponding search computation-wise plausible. Its main innovation is a strategy to approximate the global cost by combining sub-cost of multiple paths, whose computation is effectively carried out by dynamic programming. It makes sense to migrate successful computer vision algorithms to tensor platforms where developers can deploy parallel computation power with ease, and at the meanwhile, the simplified work-flow of codes (as less low-level parallel commands involved) can help the community to robustly re-produce the reported results (less local hardware configuration dependent) from other peers. Besides, tensor platforms carry now-a-days popular deep learning models for recognition purposes. Implementing the full stack system on a single platform is beneficial. In this work, we derive and implement SGM, in tensor form, based on which we build a one-shot dense 3D acquisition system using structured light, which is novel, fast and capable to produce state-of-art reconstruction results. As a new approach, it opens new possibilities as we expected. We went beyond soon. The same tensorised framework help construct a solution to patterns interfering problem, which prevents multiple one-shot structured light systems to working together. We mark the work cooping multiple structured light systems with 'beyond' throughout the paper, so that to remind the readers that the main progress we would like to present here is: the fully tensorised 3D structured light framework. 1.1 Brief the structured light system The main part of this paper describes our work building a 'one-shot' 3D acquisition system using structured light and fully implemented in tensor form. Fig. 1 shows the appearance and Fig. 2 the work-flow. Fig. 1Open in figure viewerPowerPoint Appearance of our system set-up, including: projector, camera and a laptop Fig. 2Open in figure viewerPowerPoint Flow chart of our structured light pipeline (a) Pattern generation framework, including: a. 1 – random sampling in certain region of spectrum, a. 2 – convert to spatial domain to produce pattern, a. 3 – project on object, and a. 4 – capture the object with the pattern projected on, (b) Extract disparity between the spatial pattern and camera capture, (c) 3D reconstruction result by triangulation and meshing from the disparity results Being an active illumination approach, our system is suitable for texture-less surfaces, where multi-view stereo approaches struggle. 'One-shot' property shields our system against motion that may otherwise be substantial during capturing, thus also enabling the capture of dynamic 3D, in quick succession (4D reconstruction) (cf. Section 8.3). Our system is fully designed and implemented in tensor manner, so that to deploy parallel computation power with ease. An important finding during our work is that the 'tensor shift' manoeuvre (cf. Section 6.1) access neighbours for the full-scale image in parallel, which can apply to other algorithms with similar characteristics, e.g. random fields. 1.2 Attenuation measurement To establish dense (per-pixel) disparity map therefore 3D reconstruction, we deploy SGM. Yet, the matching cannot perform well directly between projected pattern and captured image. Though theoretically one can treat a projector as an inverse camera, the projected pattern itself does not contain any photometric information. Or to say, a pattern from protector to being captured by camera is an information loss process. Empirically, this process behaves like a low-pass filtering, i.e. low frequency part seems to endure while the high frequency part lost, a blurred image. We find that by measuring the spectrum attenuation and generate patterns using according bandpass spectrum, the SGM can establish correspondence with high accuracy and robustness. Therefore, in our experiment we measure the attenuation ratio across spectrum bands before generating texture patterns. This is beneficial, as details beyond certain threshold would simply disturb the correspondence search instead of contributing (projector 'sees' these details while camera not). To make the measurement convenient, we proposed a method with invariance given epipolar geometry [4] and discrete cosine transformation (DCT) horizontal constant spectrum [5]. Details are given in Section 3. 1.3 Our projected patterns The pattern is crucial in structured light, and in general active illumination based 3D reconstruction methods. Instead of inserting handcrafted 'codes' to disambiguate sparse coordinates via their spatial surroundings [6], our pipeline retrieves 3D information by establishing dense correspondences between the captured image and projected pattern. As usual, we exploit epipolar geometry to simplify the process. The actual 3D coordinates then follow from triangulation. The pattern that we project consists of high-contrast noise, thus creating locally unique texture patterns (i.e. the 'implicit code' of the pattern is the texture itself [7]). The noise is of sufficient frequency to localise the patterns, necessary to survive the change in viewpoint between projector and camera. The spectrum of our pattern is generated randomly in a bandpass range, as a band-pass random noise pattern. It is governed by one bandwidth parameter measured previously, that steers the roughness of the pattern, so that patterns at different resolution and frequency can be generated as long as they are in proper range. More details are provided in Section 4. 1.4 Correspondence search For the correspondence search we propose tensor-SGM (tSGM) [8], innovated from SGM [3], which searches for the disparity map that minimises an approximated global cost between two images with known epipolar geometry (cf. Section 5.2). For the inter-pixel metric, we chose the census transform [9], which calculates a binary vector for each pixel encoding their topological relation regarding local neighbourhoods. Details are given in Section 5. 1.5 Derive SGM to tSGM A cornerstone of our proposed system is 'tensor shift' manoeuvre, which enables the tensorisation of SGM into tSGM [8], so that to be easily cast into parallel computing devices using tensor processing libraries (PyTorch in our work). Concrete derivation and examples are in Section 6. To demonstrate the overall performance of our system, several 3D scene reconstructions in real-world are performed (cf. 8), and a 4D reconstruction (dynamic capturing) is given in the video corresponding to this paper. We further demonstrate that the tSGM framework is not only suitable for SGM itself, but also can be easily extended into its variants; details are given in Section 8.1.1. 1.6 Experiments We perform experiments in three perspectives: (i) 'tensor shift' can lead to more innovations (cf. Section 8.1); (ii) quality of 3D reconstruction (cf. Section 8.2); and (iii) 4D reconstruction (cf. Section 8.3). 1.7 Beyond: cooping multiple structured light systems Same in tensor manner, we develop a framework to enable multiple structured light units to work simultaneously, which support applications require detail 3D from different profiles, e.g. in-situ face identification. The key is to let the camera know where the patterns come from when more than one patterns present. Then follows the reconstruction strategy: a projected pattern in combination with a camera allows for a structured light approach, which reconstructs regions illuminated by a single pattern; overlapped patterns often indicates being captured by both cameras, which allows active stereo. To identify single pattern regions versus overlaps, we watermark the different projection patterns while preserving enough textures for correspondence match. In the setup reported by this paper, two projectors and two cameras are deployed. The same mechanism can be extended to many. 1.7.1 Spectrum embedding We develop our method in spectrum embedding/watermarking, whose characteristics are beneficial: a spectrum embedding spreads over the whole spatial extent of the image, and is therefore could be detected at any part and is regardless of pattern size. The invariant mechanism in our work is based on the constant spectrum properties of DCT [10] and epipolar constrain [4]. Details are given in Section 7.2 1.7.2 Response detection and segmentation We embed the pattern identification code in DCT domain. Spectrum information in DCT can be obtained by convolution, where tensor computation platforms offer well-optimised performance (PyTorch in this work). Compared to discrete Fourier transform (DFT) [11], DCT is simpler as it abandoned the phase information. This cancels out the shift-invariant property of DFT. To compensate, we deploy the pooling function. Later, a tailored metric measurement is proposed to make the pattern detection more robust. Segmentation of the metric response is carried out by first K-Means clustering and active contour method. Concrete method is given in Section 7.5. All manipulations are in tensor manner. 1.7.3 Experiments and discussion We present experiments in both simulation and real-world practise in Section 8.4. 2 Related work As far as we know, a 3D acquisition system designed particularly to deploy tensor computation has not be seen in published works, therefore hardly related works can be found. This section will describe related works regarding 3D reconstruction to help illustrate the knowledge spectrum. A general overview of the many 3D acquisition methods proposed thus far is given by Moons et al. [6]. The pro's and con's of structured light methods are also elaborated, of which many versions have been studied; for details see [7, 12, 13]. Two main categories of structured light methods are presented: temporal encoding methods [14-16] and spatial encoding methods [7, 17, 18]. 2.1 Pattern generation Pattern design is an essential part of the structured light approach. The patterns can basically be stripes [19], grids [20] or De Bruijn patterns [21]. White-noise dot patterns have also been reported to provide good performance [12, 15, 22, 23]. 2.2 Correspondence search There is a large body of work studying the correspondence search problem [13, 24-26]. SGM proposed [3] can be considered as the state-of-the-art in corresponding search, along with its variations [27-31]. In [29], authors attribute SGM's good performance to message passing theory. Due to its individual path-processing, SGM can leverage parallel computation. A work combining SGM stereo matching with a CNN is proposed in [31]. In [27], the correspondence matching path is discussed and innovated further. 2.3 Cooping multiple structured light Cooping multi-projections is a rational attempt to extend the working range of active illumination based methods. As it is still novel, not much works reported. Several approaches, which deploy multiple projections simultaneously in one-shot manner, have been proposed [12, 22]. Sagawa et al. presents a system using random bandpass white noise pattern [22]. They train a CNN on simulated dataset to learn pattern separation. Multi-projector, multi-camera setting for capturing entire shaped human body is reported in [16]. 3 Attenuation measurement Stereo matching methods perform poorly when directly applied to structured light pairs [5], as the photometrical difference is obvious between what is projected and captured yet projector cannot catch the influence of illumination and reflection on object surface. The obvious photometrical difference results in the failure of similarity measurement across the matching image pair. Empirically, a pattern from projector to being captured by camera can be treated as an information loss process. More exactly, its behaviour is similar to a low-pass filter, i.e. the high frequency attenuates fast while low frequency part endures. Our pattern generation belongs to the spectrum-based random patterns. In this case, the information provided by the high-frequency part cannot help correspondence match but hinder it as noise. This renders an attenuation measurement necessary so that the pattern texture is generated only in the proper frequency part; therefore the projector and camera see almost the same. By measuring the spectrum attenuation in a structured light set, the range of spectrum which is suitable for generating pattern textures can be determined. To make the process simple and effective, we devise a spectrum attenuation measurement pipeline. As DCT [10] is a member of DFT family, it shares the properties of rotation. With the assumption that the attenuation is homogeneous in all directions, measuring the spectrum along any direction would provide a reasonable estimation of attenuation coefficients across spectrum domain. We first propose an invariant mechanism in the spectrum regards epipolar geometry to make the measurement simple. 3.1 Invariant mechanism The invariant mechanism is based on epipolar geometry and the constant horizontal part of DCT spectrum. Assuming the projector and camera are aligned horizontally (say along x-axis), the constant horizontal spectrum are with . It would be of the appearance as cosine function along y-axis while x component constant as shown in Fig. 3. Fig. 3Open in figure viewerPowerPoint In rectified image plane, epipolar geometry constrains the disparity/deformation is only along the baseline, the x-axis in this case. It is obvious that the patterns generated by constant horizontal spectrum remain invariant under all geometrical deformations (a) Embedding at lower spectrum, (b) Embedding at higher spectrum The spectrum marked is among the constant horizontal part. As the projector–camera pair establishes an epipolar constrain, epipolar geometry constrains the deformation only along the baseline. In our case, the disparity only changes along the x-axis. It is obvious that, the pattern generated by the constant horizontal spectrum remains invariant if the baseline in rectified image plane is horizontal, so does their spectrum. Or to say, the constant spectrum projected is observed the same while captured by the camera on the rectified image plane. For epipolar geometry, readers can refer to [4]. With the constant horizontal spectrum invariance mechanism, we therefore propose our attenuation measurement procedure. At first, one determines the original size of the pattern to be projected, often the native resolution of the projector. For the moment, the pattern is still empty in contents but only size determined. We mark the spectrum in the rectified image plane using the constant horizontal spectrum. Then, the pattern is warped back into the projector image plane. After a camera captured the projected pattern. We warp the captured image to the rectified plane and measure the attenuation coefficient. By assuming the very low-frequency part does not attenuate at all, the coefficients of attenuation across different spectrum can be retrieved. 4 Pattern generation The pattern design is important for structured light (and in general active illumination) algorithms. We vary the original approach that uses a DFT proposed in [15] to a DCT to simplify the parameterisation (i.e. no phase variable in DCT). In image compression, it has been found that reading the DCT components starting from the DC term and moving in a 'zig-zag' pattern towards the opposite corner, allows for efficient compression by truncation [32]. Our 'band-pass' random noise patterns follow a similar strategy: we start from a flat spectrum and then put DCT values outside the upper-left triangular region of the DCT domain (i.e. a triangle that has DC as one of its corners) to zero. See the right column in Fig. 4. Fig. 4Open in figure viewerPowerPoint Examples of generated band-pass random noise patterns using different r values (0.1, 0.35). The left column shows the two patterns in the spatial domain, whereas the right column shows their DCT spectrum. The constant component is set to 0 for visualisation 5 Theory for correspondence search This section discusses how to adapt census transform to structured light scenario (cf. Section 5.1) from its stereo origin, and a brief review of SGM (cf. Section 5.2). 5.1 Pixel-wise matching metric In our experiments, the photometrical differences are mainly observed in two ways: (i) projector 'sees' the correct structure of pattern yet without radiometrical interaction with the target surface, i.e. it 'sees' identically to what it projects, however the camera captures image as the ensemble effect of illumination and reflection; (ii) projector has smaller depth of field compared to cameras, in other words, projected pattern is prone to blur on the object when distance changes. In experiments the latter process seems like a low-pass filtering. To help correspondence search, the pattern for matching which the camera captured image is firstly blurred by a Gaussian filter (low-pass filtering), in order to bring the projected pattern and captured image to a similar blurry level. Depending on the scene, the optimal size of Gaussian filter varies. In our experiments, a size of 5 works well. Census transform [9] is a good method here for its well handling on significant illumination changes. An example is shown in Fig. 5. It embeds a pixel regarding its neighbourhood into a bit-vector . For any channel of , either 1 or 0 presents, meaning whether this neighbour is brighter than present pixel or not. We apply census transform to both projected pattern and camera captured image to prepare the unary cost metric. Fig. 5Open in figure viewerPowerPoint Census transform (a), (b) Structured image pair for matching, (c), (d) One channel of their census transform accordingly. Census transform describe a pixel regard to its neighbourhood as binary vector, therefore robust in illumination change 5.2 Brief review of SGM Only pixel-wise cost calculation is often ambiguous and the wrong matches can have a lower cost than the correct ones, while a global optimisation is an NP-complete problem [33] and unsolvable in tolerable time, henceforth SGM establishes image correspondence by approximating a global minimised matching cost (1)The idea of SGM is to perform path-wise optimisation along a few paths to approximate the global cost (as shown in (2)). In SGM, cost function in (1) is broke down into path-wise optimisation (2)where indicates to different paths, indicates to current pixel, d for current calculating disparity and referring to previous pixel along the path . Sixteen paths are recommended in the original paper to improve convergence as suggested in [3]. 6 Formulation: tensor-SGM In this section, we derive the full SGM pipeline into tensor form, including the census transform. 6.1 Tensor shift We begin this section with introducing 'tensor shift', which is intensively deployed in tSGM as the essential manoeuvre, to access the neighbourhood for every pixel across the image in parallel. Let us assume is a N-dimensional tensor, using to index the entries in all dimensions. are all the indexes available for tensor . If given a shift vector , will retrieve the entries shifted by and return , which satisfies (3)One shall pay attention that, 'tensor shift' consists of a re-indexing in a tensor instead of forming a new one, which prevents the effort of allocating new memory. Python-style indexing grammar is used to help illustration. 6.2 Census transform in tensor form Let us consider one channel of census transform, say a neighbour with shift regarding the kernel centre pixel for all pixels in the image. Then the response of this channel in census transform is calculated as . Obviously, this could be naturally paralleled in tensor form, as shifting the same distance for all the pixels is equivalent to shifting the whole tensor, as described in (3). In other words, extracting one channel of census transform for the whole image accomplishes in a single tensor manipulation (Fig. 6). Fig. 6Open in figure viewerPowerPoint Algorithm 1: census transform in tensor is the final census transformed tensor of original image , with the shape as . Its size would be about , where and are width and height of the input image , and is the number of channels. Pay attention that the exact size would be affected by customised padding. Calculating the unary cost in hamming distance is simple in tensor form. We denote hamming distance of two vectors and using . When this operator applies on multi-dimension (e.g. 3D) tensor, the broadcast strategy is to calculate the hamming distance over the first dimension (i.e. the channel-number dimension of previously calculated for instance) (Fig. 7). Fig. 7Open in figure viewerPowerPoint Algorithm 2: unary cost Here unary_cost is a tensor, which has the form as where d for disparity, h for height index and w for width index. 6.3 Tensor-SGM Remind that the core processing formula in SGM is as (2). To make the appearance consistent to tensor manipulation in practice, e.g. Python, we use to describe the same information, where h and w are indexes of height and width. There are three basic types of scanning paths in SGM: horizontal, vertical and diagonal. As one might notice from (2), the less easy part is how to process the terms inside min function, concerning both path manipulation and disparity manipulation , using 'tensor shift'. 6.3.1 Access spatial neighbourhood using 'tensor shift' We begin from horizontal paths, where the previous neighbour of each pixel is next to its right/left, in other words, for each pixel related cost in , its corresponding previous pixel(s) to the right/left can be retrieved as or , respectively. Similarly, for vertical paths, to access corresponding previous step pixel of each pixel would be simply or , as illustrated in Fig. 8. Fig. 8Open in figure viewerPowerPoint Examples for tSGM path and disparity access (a) Vertical path, (b) Diagonal path, (c) Combined path, (d) Disparity access by tensor shift For diagonal paths, depending on how one sweeps across the image, i.e. in vertical manner or horizontal manner, slightly different formulas could be derived. We show an example here the vertical case as shown in Fig. 8b. The previous pixels of with one step back are . Pay attention that there is one pixel without neighbour (the red pixel), so that to avoid illegal memory access. 6.3.2 Access depth neighbourhood using 'tensor shift' As shown in the first sub-figure in Fig. 8d, in disparity channel would extract the cost of identical disparity for each pixel. To describe disparity change and , as shown in the second and third sub-figures in Fig. 8d, a simple shift would do. could be retrieved using in disparity index. Similarly, could be collected as in disparity index. Again, notice that one pixel is without legal access. For , situation is a bit different, since a min function changes the size of the retrieved tensor. By assigning the minimum values from previous step as the last sub-figure in Fig. 8d, one should reshape the tensor after min function so that it can be broadcast. 6.3.3 Combine both: tensor form By combining the manipulation in Sections 6.3.1 and 6.3.2, we could write down the formula of diagonal paths as Fig. 8b for instance. Notice that tensor shifts take place in both spatial dimension(s) and disparity dimension (Fig. 9). Fig. 9Open in figure viewerPowerPoint Algorithm 3 Diagonal Path: Updating the ith Row Vertical paths and horizontal paths could be similarly derived by cancelling the spatial shift term (i.e. ). By appending a diagonal path after a vertical/horizontal path inter-changing at every row/column, another eight paths could be generated as Fig. 8c, so that the suggested 16 paths could be achieved. 7 Beyond: framework to coop multiple structured light The invariant mechanism and tensorisation leads to a framework which coops multiple structured lights to work simultaneously [5]. It can support applications which require a 3D reconstruction from multiple views, e.g. face identification for online bank service. This section aims to present our works excluded to the one-shot system but the same framework, i.e. tensorisation, as an introductory to show what else our proposed tensorisation framework can produce. To avoid confusion to the contents of the tensorised one-shot structured light, we leave the details in the original paper. The framework allows multiple structured lights working simultaneously with an acquisition strategy. A projected pattern in combination with a camera allows for a structured light approach. This is beneficial, given the weakly textured surfaces we are dealing with. Yet, where projection patterns overlap, the system automatically changes over to a multi-view approach. In order to let the system automatically detect whether a single projection versus an overlap is observed, we watermark the different projection patterns while preserving enough textures for correspondence match. In the reported system, two projectors and two cameras are deployed. Each camera–project pair consists of a one-shot structured light set and the two cameras consist of a multi-view stereo. An example of pattern identification with two patterns' presence is shown in Fig. 10 and the results in Fig. 11. Fig. 10Open in figure viewerPowerPoint