A system, comprising: one or more computer subsystems; and one or more components executed by the one or more computer subsystems, wherein the one or more components comprise: a neural network configured for determining inverted features of input images in a training set for a specimen input to the neural network, wherein the specimen is a wafer or a reticle; wherein the neural network is configured to perform deconvolution of the input images to obtain the inverted features; a forward physical model configured for reconstructing the input images from the inverted features thereby generating a set of output images corresponding to the input images in the training set, wherein the forward physical model is an optical blurring model and comprises: model parameters corresponding to imaging parameters used for generating the input images for the specimen; and a residue layer configured for determining differences between the input images in the training set and their corresponding output images in the set; wherein the one or more computer subsystems are configured for altering one or more parameters of the neural network based on the determined differences thereby training the neural network; and wherein the one or more computer subsystems are further configured to input a runtime image for the specimen or another specimen into the trained neural network, the runtime image being a blurred image such that: the trained neural network determines the inverted features for the runtime image, the forward physical model reconstructs the runtime image from the inverted features determined for the runtime image; and the residue layer determines differences between the runtime image and the reconstructed runtime image, wherein the inverted features determined for the runtime image is a sharp image which is generated by removing blur from the runtime image, and wherein the differences between the runtime image and the reconstructed runtime image are used to generate a residue image. one or more computer subsystems; and one or more components executed by the one or more computer subsystems, wherein the one or more components comprise: a neural network configured for determining inverted features of input images in a training set for a specimen input to the neural network, wherein the specimen is a wafer or a reticle; wherein the neural network is configured to perform deconvolution of the input images to obtain the inverted features; a forward physical model configured for reconstructing the input images from the inverted features thereby generating a set of output images corresponding to the input images in the training set, wherein the forward physical model is an optical blurring model and comprises: model parameters corresponding to imaging parameters used for generating the input images for the specimen; and a residue layer configured for determining differences between the input images in the training set and their corresponding output images in the set; a neural network configured for determining inverted features of input images in a training set for a specimen input to the neural network, wherein the specimen is a wafer or a reticle; wherein the neural network is configured to perform deconvolution of the input images to obtain the inverted features; a forward physical model configured for reconstructing the input images from the inverted features thereby generating a set of output images corresponding to the input images in the training set, wherein the forward physical model is an optical blurring model and comprises: model parameters corresponding to imaging parameters used for generating the input images for the specimen; and a residue layer configured for determining differences between the input images in the training set and their corresponding output images in the set; wherein the one or more computer subsystems are configured for altering one or more parameters of the neural network based on the determined differences thereby training the neural network; and wherein the one or more computer subsystems are further configured to input a runtime image for the specimen or another specimen into the trained neural network, the runtime image being a blurred image such that: the trained neural network determines the inverted features for the runtime image, the forward physical model reconstructs the runtime image from the inverted features determined for the runtime image; and the residue layer determines differences between the runtime image and the reconstructed runtime image, wherein the inverted features determined for the runtime image is a sharp image which is generated by removing blur from the runtime image, and wherein the differences between the runtime image and the reconstructed runtime image are used to generate a residue image. the trained neural network determines the inverted features for the runtime image, the forward physical model reconstructs the runtime image from the inverted features determined for the runtime image; and the residue layer determines differences between the runtime image and the reconstructed runtime image, wherein the inverted features determined for the runtime image is a sharp image which is generated by removing blur from the runtime image, and wherein the differences between the runtime image and the reconstructed runtime image are used to generate a residue image.