Patent Description:
In recent years, virtual reconstruction techniques for archaeological remains have been oriented in a preferential way to visually restore three-dimensional models.

Documents like "<NPL>et al, explore different techniques for obtaining a three-dimensional model, able to be edited by a computer program, from a plurality of photographs obtained from the real buildings.

However, these studies do not provide any method for reconstructing the buildings within the image, since they are aimed to the provision of a three-dimensional model out of bidimensional images.

Further, some work has been also done on image inpainting. Kriging models based on statistics were used by Keaomanee, Heednacram and Young Kong (<NUM>), where the input image presented a black patch requiring filling to complete the image. (<NUM>) developed a hybrid module using Recurrent Feature Reasoning and Knowledge Consistent Attention to fill large and continuous holes in images. These two previous works applied image inpainting with classical techniques instead of deep learning, which were the models used in our study.

However, all these works need to be fed with marked zones, where the missing objects need to be supplied.

<NPL>on) proposes a way of restoring bidimensional murals by using a generative adversarial network. Each layer performs different steps, including the extraction of deep image features and authenticity checking.

A solution for reconstructing damaged buildings withing complete images without it being necessary to specifically state which areas of the image to complete and how these areas should be completed is therefore sought.

The invention provides a solution for this problem by means of a method according to claim <NUM>. Preferred embodiments of the invention are defined in dependent claims.

In a first inventive aspect, the invention provides a method for obtaining an image with a restored object as defined in claim <NUM>.

This method comprises two main stage: the training stage and the processing stage. The training stage comprises the first three steps of the defined method (providing the training dataset, providing the generative adversarial network (GAN) architecture and training the GAN) and the processing stage comprises the use of the trained GAN to transform a first bidimensional image (comprising a ruined object) into a second bidimensional image (which comprises the reconstructed object).

Compared with the state of the art, the present invention provides a method which is able to infer which elements are missing or partially missing in the first bidimensional image and in which location should these elements be placed. There is no need to mark the first bidimensional image, contrary to the known inpainting methods. Further, the elements to be added are not inferred from the same first bidimensional image, contrary to the known inpainting methods.

This method is limited to the provision of a first image and the obtention of a second image which has the same size as the first image, since the machine learning is focused on identifying the missing elements and the location in the bidimensional image.

In any case, the method uses complete images, without any black patches or marks that could help the system to identify the zones with missing elements. The GAN is trained to identify both the elements and the locations where these elements should be completed. But the method does not restore the image itself, but the ruined building which is within the image.

Due to the fact of training the GAN with a training dataset comprising pairs of images, the pairs comprising two images of the same building (one restored and one partially ruined), the GAN learns to identify in the processing stage which elements are missing and the location where these elements should be placed.

In some particular embodiments, the training dataset comprises images grouped in different buildings, so that, for each building, the training data set comprises at least <NUM> different pair of images, each pair being obtained varying the perspective.

By feeding the training dataset with different perspectives of the same building, the GAN is trained to work in different perspectives. For example, for the same building, images are taken each <NUM> degrees, until completing the whole <NUM> degree range. For each <NUM> degrees, both the completed-building image and the ruined-building image are provided.

In some particular embodiments, at least part of the images of the training dataset have been created by a computer aided design program.

A computer aided design program may be used to create some of the images, since, in some cases, there is not a real picture for the restored building.

According to the invention, the step of training the generative adversarial network comprises.

For example, the segmented training uses images in which each architectonic element of the building is referenced with a colour. These images are used alongside images with the complete temples as a training guide to identify the architectonical missing elements, but in any case, there are no missing pixels. With this process, the GAN is trained to identify each type of element, and learn the location of them inside the building. The segmented training is used so that in the processing stage, there is no need to mark the first bidimensional image in any way.

In some particular embodiments, the step of training the generative adversarial network comprises.

With these steps, the accuracy is improved, taking advantage of the GAN structure, applied to this particular case.

In some particular embodiments, the step of processing the first bidimensional image by the generative adversarial network comprises.

This way of operating the first bidimensional image is especially advantageous for being processed by a generative adversarial network architecture.

In some particular embodiments, the generative adversarial network comprises.

In particular embodiments, the input block has <NUM> neurons, the first convolutional block has <NUM> neurons, the second convolutional block has <NUM> network and the third to seventh convolutional blocks have <NUM> neurons.

This structure is especially adapted to the analysis of bidimensional images, thus providing the best result in terms of machine learning.

In some particular embodiments, the discriminator network comprises.

The output layer provides a way of evaluating the quality of the image created by the generator network in a zone-by-zone basis. Each pixel of the matrix of values provides an evaluation of the quality of the building reconstruction in one of the zone (pillars, frontispiece, capitals,. Due to the fact that the method does not work with marked images or patches, this output layer provides an advantageous zone evaluator.

The scope of the invention and its embodiments is defined by the appended claims.

Accordingly, while embodiment can be modified in various ways and take on various alternative forms, specific embodiments thereof are shown in the drawings and described in detail below as examples. There is no intent to limit to the particular forms disclosed. On the contrary, all modifications, equivalents, and alternatives falling within the scope of the appended claims should be included. Elements of the example embodiments are consistently denoted by the same reference numerals throughout the drawings and detailed description where appropriate.

<FIG> shows a general approach of a method for providing an image with a restored temple according to the invention. This method has two main stages: the training stage and the processing stage. This method is particularly applied to Doric-style Greek temples.

The training stage <NUM>, which is done only once, comprises the steps of.

The processing stage <NUM>, which may be done as many times as required, comprises the following steps.

In the training stage <NUM>, the training dataset <NUM> comprises pairs of training images, each pair comprising a first training image with a complete building and a second training image, with the same building in a ruined state.

The training dataset is classified by the type of temples. For each particular temple, the training data set comprises <NUM> different pair of images, each pair being obtained each <NUM> degrees, until completing the whole <NUM> degree range. For each <NUM> degrees there is one pair of images, one with the restored temple and one with the ruined temple. By feeding the training dataset with different perspectives of the same building, the GAN is trained to work in different perspectives.

<FIG> shows an example of a pair of images. The left image <NUM> would be the image with the complete temple and the right image <NUM> would be the image with a partially ruined temple. As may be seen in this figure, both images are complete, there are not missing or deteriorated pixels.

These images of the training dataset have been created by a computer aided design program.

<FIG> show detailed views of some steps of the training stage of a particular embodiment of a method according to the invention.

In these steps, a first group of images is chosen to perform a double type of training.

<FIG> shows a first training, which is a direct training: the first training image <NUM> comprising the ruined temple is used as an input and a second training image <NUM> comprising the corresponding temple but completed is used as the output image.

<FIG> shows a second training, which is a segmented training. In this case, the input comprises both the first training image <NUM> (comprising the ruined temple) and an additional training image <NUM> where the elements are marked in the completed temple. The output will be the second training image <NUM> with the complete temple as required.

The step of training the generator network comprises.

To train the generator network, a discriminator network such as the following one is used:.

Once the generator network has been trained, as shown in the previous figures, the generator network is used to provide images with restored buildings.

When the input bidimensional image is received, the following steps are performed.

In this case, there are seven convolutional blocks, each one comprising a bidimensional convolutional layer, a batch normalization layer and leaky ReLU as the activation function with an slope of <NUM>. The first three convolutional blocks have an increasing number of neurons (<NUM>, <NUM> and <NUM>) and the rest of the blocks have all of them <NUM> neurons.

There are also seven deconvolutional blocks, each one comprising a transposed bidimensional convolutional layer, a batch normalization layer, a dropout layer and leaky ReLU as the activation function. The first three blocks have <NUM> neurons, while the last four blocks have a decreasing number of neurons (<NUM>, <NUM>, <NUM>, <NUM>).

Claim 1:
Method for obtaining an image with a restored object, the method comprising the steps of
providing a training dataset with pairs of training images, each pair comprising a first training image with a complete building and a second training image, with the same building in a ruined state
providing a generative adversarial network architecture comprising a generator network and a discriminator network
training the generative adversarial network with the training dataset and performing a machine learning process from the training dataset
providing the generative adversarial network with a first bidimensional image comprising a building in a ruined state
processing the first bidimensional image by the generative adversarial network
obtaining a second bidimensional image of the same size as the first bidimensional image, the second bidimensional image comprising a restored version of the building, the restored version of the building being produced by the trained generative adversarial network which, due to the training steps, is enabled to identify elements to be added to the building and the location where these elements are added,
wherein the step of training the generative adversarial network comprises that
a first group of pairs of images of the training dataset are used for a direct training of the generator network, where the first training image is used as an input and the second training image is used as an output;
the method being characterized in that
this first group of pairs of images of the training data set are used for a segmented training of the generator network, where the first training image is used as an input together with a segmented image, comprising an identification of different architectonic elements of the first training image.