<table>  
<tr>  
<th>

</th> </tr>  
<tr>  
<td>

_Figure 1 . Witness-Cam rig tests by SRL and Quine_

</td> </tr> </table>

The first contributions of this ​ _LADIO_DATASET_Live_Action_ will be
available in June 2017, and we expect to further expand the dataset.

  2. _**LADIO_DATASET_Advanced_3D_Reconstruction** _

  

  

This data will consist of ​ _web-collected data_ ​and include, based upon an
academic data set from INP experiments in WP3. It will include :

  1. A selection of publicly available 3D object models, 

  

  2. A set of synthetically rendered images per object model coupled with camera poses, 

  

  3. A set of web-selected real images of object model instances coupled with camera poses. 

  

  

In a) the 3D object models will be taken from sites sharing online free 3D
models e.g., like free3D.com (formerly tf3dm.com). They will be completed by
pre-computed differential geometry information (Gaussian maps, derivatives
w.r.t. the surface parameters, first and second fundamental forms, principal
curvatures etc.) which will be associated to each vertex of the surface. In
LADIO, the 3D object will be given by a set of 3D depth maps, which describes
how the original object surface is “shortened” by a perspective viewing. We
will provide a code along with the dataset for generating the depth map from
the 3D models, given a camera pose. On the other hand, b) and c) will be used
as test intensity images. In b) camera poses are known and will correspond to
ground-truth. In c) the camera poses camera poses will be determined by ​
_manually_ registering the real images to the 3D model and will be considered
as ground-truth.

This data set will allow to evaluate the performances of algorithms for
registering 2D images to 3D untextured models. In particular it will allow to
measures the degree of repeatability of the proposed features. In our case,
the repeatability of a feature is defined as the frequency with which one
detected in the depth image is found within pixels of the same location in the
corresponding intensity image.

We will also provide annotations for existing data sets (original data can not
be redistributed) , explaining to how to use them with respect to LADIO.

● IMPART datasets (cvssp.org/impart): multi-modal/multi-view datasets created
by Univ. of Surrey and Double Negative within the EU FP7 IMPART project. ●
PASCAL3D+ dataset (cvgl.stanford.edu/projects/pascal3d.html)  
  
<table>  
<tr>  
<th>

</th> </tr>  
<tr>  
<td>

_Figure 2. Multi-modal data footage and 3D reconstructions for various
indoor/outdoor scenes from IMPART datasets_

</td> </tr> </table>

The first contributions of this ​ _LADIO_DATASET_Advanced_3D_reconstruction_
will be made available in September 2017, and we expect to further expand the
dataset.

  3. _LADIO_DATASET_Multi_Body_

  

  

Fully general Multi-Body Structure from Motion is a very difficult problem,
which is very unconstrained and perhaps can’t be solved without adopting
additional constraints and priors for particular situations at hand. We will
therefore investigate several key use cases. We will be looking for the
additional constraints and priors allowing to find, formulate and solve a well
defined task.  
  
<table>  
<tr>  
<th>

</th>  
<th>

</th> </tr>  
<tr>  
<td>

</td>  
<td>

</td> </tr>  
<tr>  
<td>

_Figure 3. Independently moving objects and cameras lead to two disconnected
reconstructions (green top left and red top right), which are obtained in
independent scales (bottom left) but can be put together into a consistent
scale and meaningful relationship (bottom right). [Taken from J. Krcek. MUlti-
Body Structure from Motion. MSc thesis. CTU in Prague, 1997.]_

</td> </tr> </table>

We plan to investigate three cases that correspond to situations we
encountered in LADIO applications.

1\. Data: A few objects of similar size and importance moving around at the
same time. Task: Segment, independently reconstruct individual objects and try
to bring them into reasonable geometrical relationship, Figure 1.

Application: Basic research task leading to understanding, formulating and
testing different Multi-Body SfM approaches.

  1. Data: A main background scene with additional moving (nuisance) objects (cars, pedestrians, etc.). 

  

  

Application: Reconstructing large outdoors scenes for during extended time
periods when some objects (often distractors) are moving in the scene.

Task: Segment moving objects from the background scene and ignore them.

  2. Data A main background scene changing in time. 

  

  3. Task: Detect changes in the scene and build a 3D time dependent model representing the scene accurately at different time moments. 

  

  

Application: Reconstructing studio setups where parts of the scene are being
gradually restructures.

We will also extend POPART’s ground truth data set “ _​POPART_ _Virtual_
_Dataset -_ _Levallois_ _Town_ _Hall​_ ” with moving objects as virtual data,
corresponding to our 3 use cases.  
  
<table>  
<tr>  
<th>

</th> </tr>  
<tr>  
<td>

_Figure 4 . Levallois town hall dataset_

</td> </tr> </table>

This data set will allow to evaluate different scenarios and select the most
important one for further development.

The first contributions of this ​ _LADIO_DATASET_Multi_Body_ will be made
available in november 2017, and we expect to further expand the dataset.

#  Academic Publications

According to open access publications obligations In Horizon 2020 projects,
and in accordance with the global open innovation philosophy of LADIO project,
the academic partners of the consortium are committed to “Open access
publishing” (aka “Gold open access”) whenever this option is provided by the
venues where we must publish to reach the highest impact of our results.

Some of the most important venues in the research community are stored in the
IEEE Xplore digital library and IEEE does not provide any option for making
conference publications available as open access. However, it has become
common practice in some communities, including computer vision, to re-publish
these papers both on personal/insitute web pages and on arXiv.org. In spite of
the potential legal threat of publishing slightly different versions of works
on arXiv while transferring copyright of the final work to the IEEE, we are
following this dual approach since it is established practice in the
community.

#  Standards and metadata

The video files in the data sets are based on ARRIRAW [1] and ISO MP4 [2].
Additionally, text files included describe lens metadata and other parameters.

3D models are stored in industry standards Alembic [3] and FBX [4].

See also Deliverable 2.2, for more description about File Formats.

  1. _​http://www.arri.com/camera/alexa/workflow/working_with_arriraw/arriraw/format/_

  

  2. ​MPEG-4 Part 14 (​ISO​/​IEC​14496-14:2003) 

  

  3. _http://www.alembic.io/​_

  

  4. _​http://www.​**autodesk** ​.com/products/​ **fbx** ​/overview _

  

  

#  Data sets access and sharing

In the same spirit as the open source contributions of the project, LADIO’s
released data sets will be permanently available on _​http://ladioproject.eu​_
, the project's Github page _​https://github.com/alicevision_ and from Zenodo
_https://zenodo.org/collection/user-ladio​_ (to be created) or similar data
repository. _​_

The data sets will be released to the general public under the license of
_​Creative_ _Commons_ _Attribution-ShareAlike_ _4.0_ _International​_ ,
allowing researchers and other interested parties to exploit the data sets.

Reminder : under this license, the users are free to share and adapt the
content for any purpose, even commercially. The users must also give
appropriate credit, provide a link to the license, and indicate if changes
were made. If users remix, transform, or build upon the material, they must
distribute their contributions under the same license as the original. They
also may not apply legal terms or technological measures that legally restrict
others from doing anything the license permits.

#  Data sets reference and name

The identifier for these data sets will all be prefixed by 'LADIO_DATASET'.