Using automatic differentiation as a general framework for ptychographic reconstruction.

Coherent diffraction imaging methods enable imaging beyond lens-imposed resolution limits. In these methods, the object can be recovered by minimizing an error metric that quantifies the difference between diffraction patterns as observed, and those calculated from a present guess of the object. Efficient minimization methods require analytical calculation of the derivatives of the error metric, which is not always straightforward. This limits our ability to explore variations of basic imaging approaches. In this paper, we propose to substitute analytical derivative expressions with the automatic differentiation method, whereby we can achieve object reconstruction by specifying only the physics-based experimental forward model. We demonstrate the generality of the proposed method through straightforward object reconstruction for a variety of complex ptychographic experimental models.

Related papers:

title: A phase retrieval algorithm for shifting illumination
abstract: We propose a method of iterative phase retrieval that uses measured intensities in the diffraction plane to solve the phase problem in a way that bypasses the problem of lens aberration, leading to greatly improved spatial resolution. This method is stable, easy to implement experimentally, and can be used to view a large area of the specimen when that is desired.

title: Movable aperture lensless transmission microscopy: a novel phase retrieval algorithm.
abstract: We propose an iterative phase retrieval method that uses a series of diffraction patterns, measured only in intensity, to solve for both amplitude and phase of the image wave function over a wide field of view and at wavelength-limited resolution. The new technique requires an aperture that is scanned to two or more positions over the object wave function. A simple implementation of the method is modeled and demonstrated, showing how the algorithm uses overlapping data in real space to resolve ambiguities in the solution. The technique opens up the possibility of practical transmission lensless microscopy at subatomic resolution using electrons, x rays, or nuclear particles.

title: Phase retrieval with transverse translation diversity: a nonlinear optimization approach.
abstract: We develop and test a nonlinear optimization algorithm for solving the problem of phase retrieval with transverse translation diversity, where the diverse far-field intensity measurements are taken after translating the object relative to a known illumination pattern. Analytical expressions for the gradient of a squared-error metric with respect to the object, illumination and translations allow joint optimization of the object and system parameters. This approach achieves superior reconstructions, with respect to a previously reported technique [H. M. L. Faulkner and J. M. Rodenburg,  (2004)], when the system parameters are inaccurately known or in the presence of noise. Applicability of this method for samples that are smaller than the illumination pattern is explored.

title: An improved ptychographical phase retrieval algorithm for diffractive imaging.
abstract: The ptychographical iterative engine (or PIE) is a recently developed phase retrieval algorithm that employs a series of diffraction patterns recorded as a known illumination function is translated to a set of overlapping positions relative to a target sample. The technique has been demonstrated successfully at optical and X-ray wavelengths and has been shown to be robust to detector noise and to converge considerably faster than support-based phase retrieval methods. In this paper, the PIE is extended so that the requirement for an accurate model of the illumination function is removed.

title: Probe retrieval in ptychographic coherent diffractive imaging.
abstract: Ptychography is a coherent diffractive imaging method that uses multiple diffraction patterns obtained through the scan of a localized illumination on the specimen. Until recently, reconstruction algorithms for ptychographic datasets needed the a priori knowledge of the incident illumination. A new reconstruction procedure that retrieves both the specimen's image and the illumination profile was recently demonstrated with hard X-ray data. We present here the algorithm in greater details and illustrate its practical applicability with a visible light dataset. Improvements in the quality of the reconstruction are shown and compared to previous reconstruction techniques. Implications for future applications with other types of radiation are discussed.

title: Near-field ptychography: phase retrieval for inline holography using a structured illumination
abstract: Inline holography is a common phase-contrast imaging method which uses free-space propagation to encode the phase signal into measured intensities. However, quantitative retrieval of the sample's image remains challenging, imposing constraints on the nature of the sample or on the propagation distance. Here, we present a way of simultaneously retrieving the sample's complex-valued transmission function and the incident illumination function from near-field diffraction patterns. The procedure relies on the measurement diversity created by lateral translations of the sample with respect to a structured illumination. The reconstruction approach, in essence identical to that employed in ptychography, is applied to hard X-ray synchrotron measurements and to simulations. Compared to other inline holography techniques, we expect near-field ptychography to reduce reconstruction artefacts by factoring out wavefront imperfections and relaxing constraints on the sample's scattering properties, thus ultimately improving the robustness of propagation-based X-ray phase tomography.

title: Three-dimensional high-resolution quantitative microscopy of extended crystals.
abstract: Hard X-ray lens-less microscopy raises hopes for a non-invasive quantitative imaging, capable of achieving the extreme resolving power demands of nanoscience. However, a limit imposed by the partial coherence of third generation synchrotron sources restricts the sample size to the micrometer range. Recently, X-ray ptychography has been demonstrated as a solution for arbitrarily extending the field of view without degrading the resolution. Here we show that ptychography, applied in the Bragg geometry, opens new perspectives for crystalline imaging. The spatial dependence of the three-dimensional Bragg peak intensity is mapped and the entire data subsequently inverted with a Bragg-adapted phase retrieval ptychographical algorithm. We report on the image obtained from an extended crystalline sample, nanostructured from a silicon-on-insulator substrate. The possibility to retrieve, without transverse size restriction, the highly resolved three-dimensional density and displacement field will allow for the unprecedented investigation of a wide variety of crystalline materials, ranging from life science to microelectronics.

title: Ptychographic transmission microscopy in three dimensions using a multi-slice approach.
abstract: Generally, methods of three-dimensional imaging such as confocal microscopy and computed tomography rely on two essentials: multiple measurements (at a range of focus positions or rotations) and a weakly scattering specimen (to avoid distortion of the focal spot in the confocal microscope or to satisfy the projection approximation in tomography). Here we show that an alternative form of multi-measurement imaging, ptychography, can be extended to three dimensions and can successfully recover images in the presence of multiple scattering and when the projection approximation is not applicable. We demonstrate our technique experimentally using visible light, where it has applications in imaging thick samples such as biological tissues; however the results also have important implications for x ray and electron imaging.

title: Ptychographic X-ray computed tomography at the nanoscale
abstract: X-ray tomography is an invaluable tool in biomedical imaging. It can deliver the three-dimensional internal structure of entire organisms as well as that of single cells, and even gives access to quantitative information, crucially important both for medical applications and for basic research. Most frequently such information is based on X-ray attenuation. Phase contrast is sometimes used for improved visibility but remains significantly harder to quantify. Here we describe an X-ray computed tomography technique that generates quantitative high-contrast three-dimensional electron density maps from phase contrast information without reverting to assumptions of a weak phase object or negligible absorption. This method uses a ptychographic coherent imaging approach to record tomographic data sets, exploiting both the high penetration power of hard X-rays and the high sensitivity of lensless imaging. As an example, we present images of a bone sample in which structures on the 100 nm length scale such as the osteocyte lacunae and the interconnective canalicular network are clearly resolved. The recovered electron density map provides a contrast high enough to estimate nanoscale bone density variations of less than one per cent. We expect this high-resolution tomography technique to provide invaluable information for both the life and materials sciences.

title: 3D X-Ray Imaging of Continuous Objects beyond the Depth of Focus Limit.
abstract: X-ray ptychography is becoming the standard method for sub-30 nm imaging of thick extended samples. Available algorithms and computing power have traditionally restricted sample reconstruction to 2D slices. We build on recent progress in optimization algorithms and high performance computing to solve the ptychographic phase retrieval problem directly in 3D. Our approach addresses samples that do not fit entirely within the depth of focus of the imaging system. Such samples pose additional challenges because of internal diffraction effects within the sample. We demonstrate our approach on a computational sample modeled with 17 million complex variables.

title: A practical algorithm for the determination of phase from image and diffraction plane pictures
abstract: An algorithm is presented for the rapid solution of the phase of the complete wave function whose intensity in the diffraction and imaging planes of an imaging system are known. A proof is given showing that a defined error between the estimated function and the correct function must decrease as the algorithm iterates. The problem of uniqueness is discussed and results are presented demonstrating the power of the method.

title: Phase retrieval algorithms: a comparison.
abstract: Iterative algorithms for phase retrieval from intensity data are compared to gradient search methods. Both the problem of phase retrieval from two intensity measurements (in electron microscopy or wave front sensing) and the problem of phase retrieval from a single intensity measurement plus a non-negativity constraint (in astronomy) are considered, with emphasis on the latter. It is shown that both the error-reduction algorithm for the problem of a single intensity measurement and the Gerchberg-Saxton algorithm for the problem of two intensity measurements converge. The error-reduction algorithm is also shown to be closely related to the steepest-descent method. Other algorithms, including the input-output algorithm and the conjugate-gradient method, are shown to converge in practice much faster than the error-reduction algorithm. Examples are shown.

title: Phase retrieval by iterated projections.
abstract: Several strategies in phase retrieval are unified by an iterative "difference map" constructed from a pair of elementary projections and three real parameters. For the standard application in optics, where the two projections implement Fourier modulus and object support constraints, respectively, the difference map reproduces the "hybrid" form of Fienup's input-output map when a particular choice is made for two of the parameters. The geometric construction of the difference map illuminates the distinction between its fixed points and the recovered object, as well as the mechanism whereby the form of stagnation encountered by alternating projection schemes is avoided. When support constraints are replaced by object histogram or atomicity constraints, the difference map lends itself to crystallographic phase retrieval. Numerical experiments with synthetic data suggest that structures with hundreds of atoms can be solved.

title: Invited article: a [corrected] unified evaluation of iterative projection algorithms for phase retrieval.
abstract: Iterative projection algorithms are successfully being used as a substitute of lenses to recombine, numerically rather than optically, light scattered by illuminated objects. Images obtained computationally allow aberration-free diffraction-limited imaging and the possibility of using radiation for which no lenses exist. The challenge of this imaging technique is transferred from the lenses to the algorithms. We evaluate these new computational "instruments" developed for the phase-retrieval problem, and discuss acceleration strategies.

title: High-resolution three-dimensional partially coherent diffraction imaging.
abstract: The wave properties of light, particularly its coherence, are responsible for interference effects, which can be exploited in powerful imaging applications. Coherent diffractive imaging relies heavily on coherence and has recently experienced rapid growth. Coherent diffractive imaging recovers an object from its diffraction pattern by computational phasing with the potential of wavelength-limited resolution. Diminished coherence results in reconstructions that suffer from artefacts or fail completely. Here we demonstrate ab initio phasing of partially coherent diffraction patterns in three dimensions, while simultaneously determining the coherence properties of the illuminating wavefield. Both the dramatic improvements in image interpretability and the three-dimensional evaluation of the coherence will have broad implications for quantitative imaging of nanostructures and wavefield characterization with X-rays and electrons.

title: Reconstructing state mixtures from diffraction measurements
abstract: Progress in imaging and metrology depends on exquisite control over and comprehensive characterization of wave fields. As reflected in its name, coherent diffractive imaging relies on high coherence when reconstructing highly resolved images from diffraction intensities alone without the need for image-forming lenses. Fully coherent light can be described adequately by a single pure state. Yet partial coherence and imperfect detection often need to be accounted for, requiring statistical optics or the superposition of states. Furthermore, the dynamics of samples are increasingly the very objectives of experiments. Here we provide a general analytic approach to the characterization of diffractive imaging systems that can be described as low-rank mixed states. We use experimental data and simulations to show how the reconstruction technique compensates for and characterizes various sources of decoherence quantitatively. Based on ptychography, the procedure is closely related to quantum state tomography and is equally applicable to high-resolution microscopy, wave sensing and fluctuation measurements. As a result, some of the most stringent experimental conditions in ptychography can be relaxed, and susceptibility to imaging artefacts is reduced. Furthermore, the method yields high-resolution images of mixed states within the sample, which may include quantum mixtures or fast stationary stochastic processes such as vibrations, switching or steady flows.

title: Ptychographic overlap constraint errors and the limits of their numerical recovery using conjugate gradient descent methods.
abstract: Ptychographic coherent x-ray diffractive imaging is a form of scanning microscopy that does not require optics to image a sample. A series of scanned coherent diffraction patterns recorded from multiple overlapping illuminated regions on the sample are inverted numerically to retrieve its image. The technique recovers the phase lost by detecting the diffraction patterns by using experimentally known constraints, in this case the measured diffraction intensities and the assumed scan positions on the sample. The spatial resolution of the recovered image of the sample is limited by the angular extent over which the diffraction patterns are recorded and how well these constraints are known. Here, we explore how reconstruction quality degrades with uncertainties in the scan positions. We show experimentally that large errors in the assumed scan positions on the sample can be numerically determined and corrected using conjugate gradient descent methods. We also explore in simulations the limits, based on the signal to noise of the diffraction patterns and amount of overlap between adjacent scan positions, of just how large these errors can be and still be rendered tractable by this method.

title: Lateral position correction in ptychography using the gradient of intensity patterns.
abstract: Ptychography, a form of Coherent Diffractive Imaging, is used with short wavelengths (e.g. X-rays, electron beams) to achieve high-resolution image reconstructions. One of the limiting factors for the reconstruction quality is the accurate knowledge of the illumination probe positions. Recently, many advances have been made to relax the requirement for the probe positions accuracy. Here, we analyse and demonstrate a straightforward approach that can be used to correct the probe positions with sub-pixel accuracy. Simulations and experimental results with visible light are presented in this work.

title: Maximum-likelihood refinement for coherent diffractive imaging
abstract: We introduce the application of maximum-likelihood (ML) principles to the image reconstruction problem in coherent diffractive imaging. We describe an implementation of the optimization procedure for ptychography, using conjugate gradients and including preconditioning strategies, regularization and typical modifications of the statistical noise model. The optimization principle is compared to a difference map reconstruction algorithm. With simulated data important improvements are observed, as measured by a strong increase in the signal-to-noise ratio. Significant gains in resolution and sensitivity are also demonstrated in the ML refinement of a reconstruction from experimental x-ray data. The immediate consequence of our results is the possible reduction of exposure, or dose, by up to an order of magnitude for a reconstruction quality similar to iterative algorithms currently in use.

title: Noise models for low counting rate coherent diffraction imaging.
abstract: Coherent diffraction imaging (CDI) is a lens-less microscopy method that extracts the complex-valued exit field from intensity measurements alone. It is of particular importance for microscopy imaging with diffraction set-ups where high quality lenses are not available. The inversion scheme allowing the phase retrieval is based on the use of an iterative algorithm. In this work, we address the question of the choice of the iterative process in the case of data corrupted by photon or electron shot noise. Several noise models are presented and further used within two inversion strategies, the ordered subset and the scaled gradient. Based on analytical and numerical analysis together with Monte-Carlo studies, we show that any physical interpretations drawn from a CDI iterative technique require a detailed understanding of the relationship between the noise model and the used inversion method. We observe that iterative algorithms often assume implicitly a noise model. For low counting rates, each noise model behaves differently. Moreover, the used optimization strategy introduces its own artefacts. Based on this analysis, we develop a hybrid strategy which works efficiently in the absence of an informed initial guess. Our work emphasises issues which should be considered carefully when inverting experimental data.

title: Applications of algorithmic differentiation to phase retrieval algorithms.
abstract: In this paper, we generalize the techniques of reverse-mode algorithmic differentiation to include elementary operations on multidimensional arrays of complex numbers. We explore the application of the algorithmic differentiation to phase retrieval error metrics and show that reverse-mode algorithmic differentiation provides a framework for straightforward calculation of gradients of complicated error metrics without resorting to finite differences or laborious symbolic differentiation.

title: Phase Retrieval via Wirtinger Flow: Theory and Algorithms
abstract: We study the problem of recovering the phase from magnitude measurements; specifically, we wish to reconstruct a complex-valued signal x ∈ ℂn about which we have phaseless samples of the form yr = |〈ar, x〉|2, r = 1, ..., m (knowledge of the phase of these samples would yield a linear system). This paper develops a nonconvex formulation of the phase retrieval problem as well as a concrete solution algorithm. In a nutshell, this algorithm starts with a careful initialization obtained by means of a spectral method, and then refines this initial estimate by iteratively applying novel update rules, which have low computational complexity, much like in a gradient descent scheme. The main contribution is that this algorithm is shown to rigorously allow the exact retrieval of phase information from a nearly minimal number of random measurements. Indeed, the sequence of successive iterates provably converges to the solution at a geometric rate so that the proposed scheme is efficient both in terms of computational and data resources. In theory, a variation on this scheme leads to a near-linear time algorithm for a physically realizable model based on coded diffraction patterns. We illustrate the effectiveness of our methods with various experiments on image data. Underlying our analysis are insights for the analysis of nonconvex optimization schemes that may have implications for computational problems beyond phase retrieval.

title: Reshaped Wirtinger Flow for Solving Quadratic System of Equations
abstract: We study the problem of recovering a vector $\bx\in \bbR^n$ from its magnitude measurements $y_i=|\langle \ba_i, \bx\rangle|, i=1,..., m$. Our work is along the line of the Wirtinger flow (WF) approach , which solves the problem by minimizing a nonconvex loss function via a gradient algorithm and can be shown to converge to a global optimal point under good initialization. In contrast to the smooth loss function used in WF, we adopt a nonsmooth but lower-order loss function, and design a gradient-like algorithm (referred to as reshaped-WF). We show that for random Gaussian measurements, reshaped-WF enjoys geometric convergence to a global optimal point as long as the number $m$ of measurements is at the order of $\cO(n)$, where $n$ is the dimension of the unknown $\bx$. This improves the sample complexity of WF, and achieves the same sample complexity as truncated-WF \citet{chen2015solving} but without truncation at gradient step. Furthermore, reshaped-WF costs less computationally than WF, and runs faster numerically than both WF and truncated-WF. Bypassing higher-order variables in the loss function and truncations in the gradient loop, analysis of reshaped-WF is simplified.

title: Conjugate gradient method for phase retrieval based on the Wirtinger derivative.
abstract: A conjugate gradient Wirtinger flow (CG-WF) algorithm for phase retrieval is proposed in this paper. It is shown that, compared with recently reported Wirtinger flow and its modified methods, the proposed CG-WF algorithm is able to dramatically accelerate the convergence rate while keeping the dominant computational cost of each iteration unchanged. We numerically illustrate the effectiveness of our method in recovering 1D Gaussian signals and 2D natural color images under both Gaussian and coded diffraction pattern models.

title: Nonlinear Optimization Algorithm for Partially Coherent Phase Retrieval and Source Recovery
abstract: We propose a new algorithm for recovering both complex field (phase and amplitude) and source distribution (illumination spatial coherence) from a stack of intensity images captured through focus. The joint recovery is formulated as a nonlinear least-square-error optimization problem, which is solved iteratively by a modified Gauss-Newton method. We derive the gradient and Hessian of the cost function and show that our second-order optimization approach outperforms previously proposed phase retrieval algorithms, for datasets taken with both coherent and partially coherent illumination. The method is validated experimentally in a commercial microscope with both Köhler illumination and a programmable light-emitting diode dome.

title: Efficient Algorithms for Ptychographic Phase Retrieval
abstract: Ptychography is an emerging non-crystalline diffractive imaging technique that can potentially reach diffraction limited resolution without the need for high resolution lenses. To achieve high resolution one must solve a phase-retrieval inverse problem using the diffraction patterns of many partially overlapping subimage frames. We examine the mathematical formulation of the ptychographic phase retrieval problem, and analyze some of the existing methods for solving the inverse problem. We also discuss a number of practical techniques that can improve the efficiency and robustness of numerical algorithms for solving the ptychographic phase retrieval problem.

title: Fast Deterministic Ptychographic Imaging Using X-Rays
abstract: We present a deterministic approach to the ptychographic retrieval of the wave at the exit surface of a specimen of condensed matter illuminated by X-rays. The method is based on the solution of an overdetermined set of linear equations, and is robust to measurement noise. The set of linear equations is efficiently solved using the conjugate gradient least-squares method implemented using fast Fourier transforms. The method is demonstrated using a data set obtained from a gold–chromium nanostructured test object. It is shown that the transmission function retrieved by this linear method is quantitatively comparable with established methods of ptychography, with a large decrease in computational time, and is thus a good candidate for real-time reconstruction.

title: Further improvements to the ptychographical iterative engine
abstract: Ptychography is a form of phase imaging that uses iterative algorithms to reconstruct an image of a specimen from a series of diffraction patterns. It is swiftly developing into a mainstream technique, with a growing list of applications across a range of imaging modalities. As the field has advanced, numerous reconstruction algorithms have been proposed, yet the early approaches have not seen major improvement and remain popular. In this paper, we revisit the first such algorithm, the ptychographical iterative engine (PIE), and show how a simple revision and powerful extension can deliver an order of magnitude speed increase and handle difficult data sets where the original version fails completely.

title: Proximal Heterogeneous Block Implicit-Explicit Method and Application to Blind Ptychographic Diffraction Imaging
abstract: We propose a general alternating minimization algorithm for nonconvex optimization problems with separable structure and nonconvex coupling between blocks of variables. To fix our ideas, we apply the methodology to the problem of blind ptychographic imaging. Compared to other schemes in the literature, our approach differs in two ways: (i) it is posed within a clear mathematical framework with practical verifiable assumptions, and (ii) under the given assumptions, it is provably convergent to critical points. A numerical comparison of our proposed algorithm with the current state of the art on simulated and experimental data validates our approach and points toward directions for further improvement.

title: TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
abstract: TensorFlow is an interface for expressing machine learning algorithms, and an implementation for executing such algorithms. A computation expressed using TensorFlow can be executed with little or no change on a wide variety of heterogeneous systems, ranging from mobile devices such as phones and tablets up to large-scale distributed systems of hundreds of machines and thousands of computational devices such as GPU cards. The system is flexible and can be used to express a wide variety of algorithms, including training and inference algorithms for deep neural network models, and it has been used for conducting research and for deploying machine learning systems into production across more than a dozen areas of computer science and other fields, including speech recognition, computer vision, robotics, information retrieval, natural language processing, geographic information extraction, and computational drug discovery. This paper describes the TensorFlow interface and an implementation of that interface that we have built at Google. The TensorFlow API and a reference implementation were released as an open-source package under the Apache 2.0 license in November, 2015 and are available at www.tensorflow.org.

title: Automatic differentiation in PyTorch
abstract: In this article, we describe an automatic differentiation module of PyTorch — a library designed to enable rapid research on machine learning models. It builds upon a few projects, most notably Lua Torch, Chainer, and HIPS Autograd , and provides a high performance environment with easy access to automatic differentiation of models executed on different devices (CPU and GPU). To make prototyping easier, PyTorch does not follow the symbolic approach used in many other deep learning frameworks, but focuses on differentiation of purely imperative programs, with a focus on extensibility and low overhead. Note that this preprint is a draft of certain sections from an upcoming paper covering all PyTorch features.

title: Modeling, Inference and Optimization With Composable Differentiable Procedures
abstract: This thesis presents five contributions to machine learning, with themes of differentiability and Bayesian inference. We present Firefly Monte Carlo, an auxiliary variable Markov chain Monte Carlo algorithm that only queries a potentially small subset of data at each iteration yet simulates from the exact posterior distribution. We describe the design and implementation of Autograd, a software package for efficiently computing derivatives of functions written in Python/Numpy using reverse accumulation mode differentiation. Using Autograd, we develop a convolutional neural network that takes arbitrary graphs, such as organic molecules, as input. This generalizes standard molecular feature representations and allows end-to-end adaptation of the feature extraction pipeline to particular tasks. We show how to compute gradients of cross-validation loss with respect to hyperparameters of learning algorithms, with both time and memory efficiency, by chaining gradients backwards through an exactly reversed optimization procedure. Finally, by accounting for the entropy destroyed by optimization, we show that early stopping and ensembling, popular tricks for avoiding overfitting, can be interpreted as variational Bayesian inference.

title: Distributed Automatic Differentiation for Ptychography
abstract: Synchrotron radiation light source facilities are leading the way to ultrahigh resolution X-ray imaging. High resolution imaging is essential to understanding the fundamental structure and interaction of materials at the smallest length scale possible. Diffraction based methods achieve nanoscale imaging by replacing traditional objective lenses by pixelated area detectors and computational image reconstruction. Among these methods, ptychography is quickly becoming the standard for sub-30 nanometer imaging of extended samples, but at the expense of increasingly high data rates and volumes. This paper presents a new distributed algorithm for solving the ptychographic image reconstruction problem based on automatic differentiation. Input datasets are subdivided between multiple graphics processing units (GPUs); each subset of the problem is then solved either entirely independent of other subsets (asynchronously) or through sharing gradient information with other GPUs (synchronously). The algorithm was evaluated on simulated and real data acquired at the Advanced Photon Source, scaling up to 192 GPUs. The synchronous variant of our method outperformed an existing multi-GPU implementation in terms of accuracy while running at a comparable execution time.

title: ADP: Automatic differentiation ptychography
abstract: Ptychography is an imaging technique which aims to recover the complex-valued exit wavefront of an object from a set of its diffraction pattern magnitudes. Ptychography is one of the most popular techniques for sub-30 nanometer imaging as it does not suffer from the limitations of typical lens based imaging techniques. The object can be reconstructed from the captured diffraction patterns using iterative phase retrieval algorithms. Over time many algorithms have been proposed for iterative reconstruction of the object based on manually derived update rules. In this paper, we adapt automatic differentiation framework to solve practical and complex ptychographic phase retrieval problems and demonstrate its advantages in terms of speed, accuracy, adaptability and generalizability across different scanning techniques.

title: Automatic differentiation in machine learning: a survey
abstract: Derivatives, mostly in the form of gradients and Hessians, are ubiquitous in machine learning. Automatic differentiation (AD), also called algorithmic differentiation or simply “auto-diff”, is a family of techniques similar to but more general than backpropagation for efficiently and accurately evaluating derivatives of numeric functions expressed as computer programs. AD is a small but established field with applications in areas including computational fluid dynamics, atmospheric sciences, and engineering design optimization. Until 
very recently, the fields of machine learning and AD have largely been unaware of each other and, in some cases, have independently discovered each other’s results. Despite its 
relevance, general-purpose AD has been missing from the machine learning toolbox, a situation slowly changing with its ongoing adoption under the names “dynamic computational 
graphs” and “differentiable programming”. We survey the intersection of AD and machine learning, cover applications where AD has direct relevance, and address the main imple- 
mentation techniques. By precisely defining the main differentiation techniques and their interrelationships, we aim to bring clarity to the usage of the terms “autodiff”, “automatic differentiation”, and “symbolic differentiation” as these are encountered more and more in machine learning settings.

title: A Hitchhiker’s Guide to Automatic Differentiation
abstract: This article provides an overview of some of the mathematical principles of Automatic Differentiation (AD). In particular, we summarise different descriptions of the Forward Mode of AD, like the matrix-vector product based approach, the idea of lifting functions to the algebra of dual numbers, the method of Taylor series expansion on dual numbers and the application of the push-forward operator, and explain why they all reduce to the same actual chain of computations.We further give a short mathematical description of some methods of higher-order Forward AD and, at the end of this paper, briefly describe the Reverse Mode of Automatic Differentiation.

title: Fourier ptychographic reconstruction using Wirtinger flow optimization.
abstract: Recently Fourier Ptychography (FP) has attracted great attention, due to its marked effectiveness in leveraging snapshot numbers for spatial resolution in large field-of-view imaging. To acquire high signal-to-noise-ratio (SNR) images under angularly varying illuminations for subsequent reconstruction, FP requires long exposure time, which largely limits its practical applications. In this paper, based on the recently reported Wirtinger flow algorithm, we propose an iterative optimization framework incorporating phase retrieval and noise relaxation together, to realize FP reconstruction using low SNR images captured under short exposure time. Experiments on both synthetic and real captured data validate the effectiveness of the proposed reconstruction method. Specifically, the proposed technique could save ~ 80% exposure time to achieve similar retrieval accuracy compared to the conventional FP. Besides, we have released our source code for non-commercial use.

title: Alternating direction methods for classical and ptychographic phase retrieval
abstract: In this paper, we show how the augmented Lagrangian alternating direction method (ADM) can be used to solve both the classical and ptychographic phase retrieval problems. We point out the connection between ADM and projection algorithms such as the hybrid input–output algorithm, and compare its performance against standard algorithms for phase retrieval on a number of test images. Our computational experiments show that ADM appears to be less sensitive to the choice of relaxation parameters, and it usually outperforms the existing techniques for both the classical and ptychographic phase retrieval problems.

title: Reconstruction of the support of an object from the support of its autocorrelation
abstract: The phase-retrieval problem consists of the reconstruction of an object from the modulus of its Fourier transform or, equivalently, from its autocorrelation. This paper describes a number of results relating to the reconstruction of the support of an object from the support of its autocorrelation. Methods for reconstructing the object’s support are given for objects whose support is convex and for certain objects consisting of collections of distinct points. The uniqueness of solutions is discussed. In addition, for the objects consisting of collections of points, a simple method is shown for completely reconstructing the object functions.

title: A complex gradient operator and its applica-tion in adaptive array theory
abstract: The problem of minimising a real scalar quantity (for example array output power, or mean square error) as a function of a complex vector (the set of weights) frequently arises in adaptive array theory. A complex gradient operator is defined in the paper for this purpose and its use justified. Three examples of its application to array theory problems are given.

title: Unconstrained Optimization of Real Functions in Complex Variables
abstract: Nonlinear optimization problems in complex variables are frequently encountered in applied mathematics and engineering applications such as control theory, signal processing, and electrical engineering. Optimization of these problems often requires a first- or second-order approximation of the objective function to generate a new step or descent direction. However, such methods cannot be applied to real functions of complex variables because they are necessarily nonanalytic in their argument, i.e., the Taylor series expansion in their argument alone does not exist. To overcome this problem, the objective function is usually redefined as a function of the real and imaginary parts of its complex argument so that standard optimization methods can be applied. However, this approach may needlessly disguise any inherent structure present in the derivatives of such complex problems. Although little known, it is possible to construct an expansion of the objective function in its original complex variables by noti...

title: Adam: A Method for Stochastic Optimization
abstract: We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.

title: Globally convergent image reconstruction for emission tomography using relaxed ordered subsets algorithms
abstract: We present two types of globally convergent relaxed ordered subsets (OS) algorithms for penalized-likelihood image reconstruction in emission tomography: modified block sequential regularized expectation-maximization (BSREM) and relaxed OS separable paraboloidal surrogates (OS-SPS). The global convergence proof of the existing BSREM (De Pierro and Yamagishi, 2001) required a few a posteriori assumptions. By modifying the scaling functions of BSREM, we are able to prove the convergence of the modified BSREM under realistic assumptions. Our modification also makes stepsize selection more convenient. In addition, we introduce relaxation into the OS-SPS algorithm (Erdogan and Fessler, 1999) that otherwise would converge to a limit cycle. We prove the global convergence of diagonally scaled incremental gradient methods of which the relaxed OS-SPS is a special case; main results of the proofs are from (Nedic and Bertsekas, 2001) and (Correa and Lemarechal, 1993). Simulation results showed that both new algorithms achieve global convergence yet retain the fast initial convergence speed of conventional unrelaxed ordered subsets algorithms.

title: Effiicient BackProp
abstract: The convergence of back-propagation learning is analyzed so as to explain common phenomenon observed by practitioners. Many undesirable behaviors of backprop can be avoided with tricks that are rarely exposed in serious technical publications. This paper gives some of those tricks, and offers explanations of why they work. Many authors have suggested that second-order optimization methods are advantageous for neural net training. It is shown that most “classical” second-order methods are impractical for large neural networks. A few methods are proposed that do not have these limitations.

title: An overview of gradient descent optimization algorithms
abstract: Gradient descent optimization algorithms, while increasingly popular, are often used as black-box optimizers, as practical explanations of their strengths and weaknesses are hard to come by. This article aims to provide the reader with intuitions with regard to the behaviour of different algorithms that will allow her to put them to use. In the course of this overview, we look at different variants of gradient descent, summarize challenges, introduce the most common optimization algorithms, review architectures in a parallel and distributed setting, and investigate additional strategies for optimizing gradient descent.

title: Accelerated Wirtinger Flow: A fast algorithm for ptychography
abstract: This paper presents a new algorithm, Accelerated Wirtinger Flow (AWF), for ptychographic image reconstruction from phaseless diffraction pattern measurements. AWF is based on combining Nesterov's acceleration approach with Wirtinger gradient descent. Theoretical results enable prespecification of all AWF algorithm parameters, with no need for computationally-expensive line searches and no need for manual parameter tuning. AWF is evaluated in the context of simulated X-ray ptychography, where we demonstrate fast convergence and low per-iteration computational complexity. We also show examples where AWF reaches higher image quality with less computation than classical algorithms. AWF is also shown to have robustness to noise and probe misalignment.

title: Efficient subpixel image registration algorithms.
abstract: Three new algorithms for 2D translation image registration to within a small fraction of a pixel that use nonlinear optimization and matrix-multiply discrete Fourier transforms are compared. These algorithms can achieve registration with an accuracy equivalent to that of the conventional fast Fourier transform upsampling approach in a small fraction of the computation time and with greatly reduced memory requirements. Their accuracy and computation time are compared for the purpose of evaluating a translation-invariant error metric.

title: Introduction to Fourier optics
abstract: The second edition of this respected text considerably expands the original and reflects the tremendous advances made in the discipline since 1968. All material has been thoroughly updated and several new sections explore recent progress in important areas, such as wavelength modulation, analog information processing, and holography. Fourier analysis is a ubiquitous tool with applications in diverse areas of physics and engineering. This book explores these applications in the field of optics with a special emphasis on applications to diffraction, imaging, optical data processing, and holography. This book can be used as a textbook to satisfy the needs of several different types of courses, and it is directed toward both engineers ad physicists. By varying the emphasis on different topics and specific applications, the book can be used successfully in a wide range of basic Fourier Optics or Optical Signal Processing courses.

title: X-ray near-field ptychography for optically thick specimens
abstract: Inline holography, like other lensless imaging methods, circumvents limitations of x-ray optics through an a posteriori phase-retrieval step. However, phase retrieval for optically thick, i.e., strongly absorbing and phase shifting, specimens remains challenging. In this paper, we demonstrate that near-field ptychography can be used to efficiently perform phase retrieval on a uranium sphere with a diameter of about 46  μm, which acts as an optically thick sample. This particular sample was not accessible by inline holography previously. The reconstruction is based on a statistical model and incorporates partial coherence by decomposing the illumination into coherent modes. Furthermore, we observe that phase vortices, which can occur as artifacts during the reconstruction, pose a greater challenge than in far-field methods. We expect that the methods described in this paper will allow production of reliable phase maps of samples which cannot be accessed by inline holography.

title: Near-field ptychography using lateral and longitudinal shifts
abstract: Image reconstruction of in-line holography depends crucially on the probing wave front used to illuminate an object. Aberrations inherent to the illumination can mix with the features imposed by the object. Conventional raw data processing methods rely on the division of the measured hologram by the intensity profile of the probe and are not able to fully eliminate artifacts caused by the illumination. Here we present a generalized ptychography approach to simultaneously reconstruct object and probe in the optical near-field. Combining the ideas of ptychographic lateral shifts of the object with variations of the propagation distance by longitudinal shifts, simultaneous reconstruction of object and probe was achieved equally well for a highly aberrated and a mildly disturbed probe without the need for an additional wave front diffuser. The method overcomes the image deterioration by a non-ideal probe and at the same time any restrictions due to linearization of the object's transmission function or the Fresnel propagator. The method is demonstrated experimentally using visible light and hard x-rays, in both parallel beam and cone beam geometry, which is relevant for high resolution x-ray imaging. It also opens up a new approach to characterize extended wave fronts by phase retrieval.

title: Measuring Three-Dimensional Strain and Structural Defects in a Single InGaAs Nanowire Using Coherent X-ray Multiangle Bragg Projection Ptychography.
abstract: III-As nanowires are candidates for near-infrared light emitters and detectors that can be directly integrated onto silicon. However, nanoscale to microscale variations in structure, composition, and strain within a given nanowire, as well as variations between nanowires, pose challenges to correlating microstructure with device performance. In this work, we utilize coherent nanofocused X-rays to characterize stacking defects and strain in a single InGaAs nanowire supported on Si. By reconstructing diffraction patterns from the 21̅1̅0 Bragg peak, we show that the lattice orientation varies along the length of the wire, while the strain field along the cross-section is largely unaffected, leaving the band structure unperturbed. Diffraction patterns from the 011̅0 Bragg peak are reproducibly reconstructed to create three-dimensional images of stacking defects and associated lattice strains, revealing sharp planar boundaries between different crystal phases of wurtzite (WZ) structure that contribute to charge carrier scattering. Phase retrieval is made possible by developing multiangle Bragg projection ptychography (maBPP) to accommodate coherent nanodiffraction patterns measured at arbitrary overlapping positions at multiple angles about a Bragg peak, eliminating the need for scan registration at different angles. The penetrating nature of X-ray radiation, together with the relaxed constraints of maBPP, will enable the in operando imaging of nanowire devices.

title: High-resolution three-dimensional structural microscopy by single-angle Bragg ptychography.
abstract: Coherent X-ray microscopy by phase retrieval of Bragg diffraction intensities enables lattice distortions within a crystal to be imaged at nanometre-scale spatial resolutions in three dimensions. While this capability can be used to resolve structure-property relationships at the nanoscale under working conditions, strict data measurement requirements can limit the application of current approaches. Here, we introduce an efficient method of imaging three-dimensional (3D) nanoscale lattice behaviour and strain fields in crystalline materials with a methodology that we call 3D Bragg projection ptychography (3DBPP). This method enables 3D image reconstruction of a crystal volume from a series of two-dimensional X-ray Bragg coherent intensity diffraction patterns measured at a single incident beam angle. Structural information about the sample is encoded along two reciprocal-space directions normal to the Bragg diffracted exit beam, and along the third dimension in real space by the scanning beam. We present our approach with an analytical derivation, a numerical demonstration, and an experimental reconstruction of lattice distortions in a component of a nanoelectronic prototype device.

title: Absolute Uniqueness of Phase Retrieval with Random Illumination
abstract: Random illumination is proposed to enforce absolute uniqueness and resolve all types of ambiguity, trivial or nontrivial, in phase retrieval. Almost sure irreducibility is proved for any complex-valued object whose support set has rank 2. While the new irreducibility result can be viewed as a probabilistic version of the classical result by Bruck, Sodin and Hayes, it provides a novel perspective and an effective method for phase retrieval. In particular, almost sure uniqueness, up to a global phase, is proved for complex-valued objects under general two-point conditions. Under a tight sector constraint absolute uniqueness is proved to hold with probability exponentially close to unity as the object sparsity increases. Under a magnitude constraint with random amplitude illumination, uniqueness modulo global phase is proved to hold with probability exponentially close to unity as object sparsity increases. For general complex-valued objects without any constraint, almost sure uniqueness up to global phase is established with two sets of Fourier magnitude data under two independent illuminations. Numerical experiments suggest that random illumination essentially alleviates most, if not all, numerical problems commonly associated with the standard phasing algorithms. (Some figures may appear in colour only in the online journal)

title: Strain in a silicon-on-insulator nanostructure revealed by 3D x-ray Bragg ptychography
abstract: Progresses in the design of well-defined electronic band structure and dedicated functionalities rely on the high control of complex architectural device nano-scaled structures. This includes the challenging accurate description of strain fields in crystalline structures, which requires non invasive and three-dimensional (3D) imaging methods. Here, we demonstrate in details how x-ray Bragg ptychography can be used to quantify in 3D a displacement field in a lithographically patterned silicon-on-insulator structure. The image of the crystalline properties, which results from the phase retrieval of a coherent intensity data set, is obtained from a well-controlled optimized process, for which all steps are detailed. These results confirm the promising perspectives of 3D Bragg ptychography for the investigation of complex nano-structured crystals in material science.