# Roadmap on Label-Free Super-Resolution Imaging

Vasily N Astratov, Yair Ben Sahel, Yonina C Eldar, Luzhe Huang, Aydogan Ozcan, Nikolay Zheludev, Junxiang Zhao, Zachary Burns, Zhaowei Liu, Evgenii Narimanov, et al.

► To cite this version:

Vasily N Astratov, Yair Ben Sahel, Yonina C Eldar, Luzhe Huang, Aydogan Ozcan, et al. Roadmap on Label-Free Super-Resolution Imaging. Laser and Photonics Reviews, 2023, 17 (12), pp.2200029. 10.1002/lpor.202200029 . hal-04355087

HAL Id: hal-04355087

https://univ-rennes.hal.science/hal-04355087v1

Submitted on 20 Dec 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

# Roadmap on Label-Free Super-Resolution Imaging

Vasily N. Astratov,\* Yair Ben Sahel, Yonina C. Eldar, Luzhe Huang, Aydogan Ozcan, Nikolay Zheludev, Junxiang Zhao, Zachary Burns, Zhaowei Liu, Evgenii Narimanov, Neha Goswami, Gabriel Popescu, Emanuel Pfitzner, Philipp Kukura, Yi-Teng Hsiao, Chia-Lung Hsieh, Brian Abbey, Alberto Diaspro, Aymeric LeGratiet, Paolo Bianchini, Natan T. Shaked, Bertrand Simon, Nicolas Verrier, Matthieu Debailleul, Olivier Haeberlé, Sheng Wang, Mengkun Liu, Yeran Bai, Ji-Xin Cheng, Behjat S. Kariman, Katsumasa Fujita, Moshe Sinvani, Zeev Zalevsky, Xiangping Li, Guan-Jie Huang, Shi-Wei Chu, Omer Tzang, Dror Hershkovitz, Ori Cheshnovsky, Mikko J. Huttunen, Stefan G. Stanciu, Vera N. Smolyaninova, Igor I. Smolyaninov, Ulf Leonhardt, Sahar Sahebdivan, Zengbo Wang, Boris Luk'yanchuk, Limin Wu, Alexey V. Maslov, Boya Jin, Constantin R. Simovski, Stephane Perrin, Paul Montgomery, and Sylvain Lecler

Label-free super-resolution (LFSR) imaging relies on light-scattering processes in nanoscale objects without a need for fluorescent (FL) staining required in super-resolved FL microscopy. The objectives of this Roadmap are to present a comprehensive vision of the developments, the state-of-the-art in this field, and to discuss the resolution boundaries and hurdles that need to be overcome to break the classical diffraction limit of the label-free imaging. The scope of this Roadmap spans from the advanced interference detection techniques, where the diffraction-limited lateral resolution is combined with unsurpassed axial and temporal resolution, to techniques with true lateral super-resolution capability that are based on understanding resolution as an information science problem, on using novel structured illumination, near-field scanning, and nonlinear optics approaches, and on designing superlenses based on nanoplasmonics, metamaterials, transformation optics, and microsphere-assisted approaches. To this end, this Roadmap brings under the same umbrella researchers from the physics and biomedical optics communities in which such studies have often been developing separately. The ultimate intent of this paper is to create a vision for the current and future developments of LFSR imaging based on its physical mechanisms and to create a great opening for the series of articles in this field.

## Foreword

Imaging is one of the central services that the physical sciences supply to the biosciences. As well as optical microscopy, the subject of this Roadmap, we can add magnetic resonance imaging, X-ray computer tomography, electron microscopy (EM), atomic force microscopy (AFM), and scanning near field optical microscopy (SNOM). It is not an exaggeration to say that the physical sciences are the handmaiden of the biosciences a hugely important function often hidden by more newsworthy aspects such as astronomy. This is a message worth emphasising to our political masters when making the case for supporting our discipline.

Optical microscopy is by far the most venerable of the imaging techniques in use. Its convenience of use and benign treatment of specimens often make it the tool of choice. Frustratingly much of biology, particularly the more modern aspects, exists on a length scale beyond the reach of conventional microscopes but

the virtues alluded to above mean that over the past decades there has been huge effort overcome the resolution problem. Although EM, AFM, and SNOM are capable of high resolution they all have drawbacks relative to the optical microscope, whether it be of speed, convenience, or specimen damage.

The Abbé diffraction limit which restricts resolution to the threshold of many of the treasures of the biosciences, was broken first by exploitation of fluorescent dye markers. Label free imaging is more challenging but hopes were raised when the shibboleth of the Abbé law was shown to be a practical limit, not an absolute one. This was a consequence of detailed investigation of the strange properties of materials with negative refractive

<figure><img src="image_3.png" /></figure>

The ORCID identification number(s) for the author(s) of this article can be found under https://doi.org/10.1002/lpor.202200029

© 2023 The Authors. Laser & Photonics Reviews published by Wiley-VCH GmbH. This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

DOI: 10.1002/lpor.202200029

<footnote>\*E-mail: astratov@charlotte.edu</footnote>

<footnote>\#The complete list of author affiliations can be found in the Addendum, at the end of the manuscript.</footnote>

<footnote>†Deceased</footnote>
indices. First raised by Veselago as a theoretical possibility, negative refraction became a reality with the advent of metamaterials at the turn of the century. Subsequent refinement of our understanding of lenses based on negative refraction revealed their potential for super-resolution. Key to their properties are the permittivity and permeability which must both be negative: this creates a strange but transparent material contrasted with metals which have only a negative permittivity and are not transparent. However both support surface states: an electrical one in the case of metals, known as a surface plasmon. For negative refraction there are two surface states: one with predominantly electrical character like a metal and the other with predominantly magnetic character which is rarely if ever seen in naturally occurring materials. It is the surface states that give rise to the super-resolution characteristics of negative refracting materials. In fact as many of the articles in this Roadmap show, metals can sometimes substitute for a truly negatively refracting material in super resolving systems.

The close relationship of negative refraction to plasmonic materials connects with a wider interest in plasmonics and the abil-

ity of plasmons formed on suitably structured surfaces to focus light into an area whose size is determined by the plasmon wavelength rather than that of the free space wavelength. Early work by Ebbesen demonstrated how light can be gathered into such a small area as to be able to squeeze through holes in silver a mere tens of nanometres across. As our understanding of negative refraction and its connection to surface resonances grew exploitation for imaging was natural consequence. Application of these ideas is reported in this Roadmap in the form of structured illumination and the development of hyperbolic materials with their positive/negative hybrid structure and their unique ability to propagate sub-wavelength radiation.

The several areas explored in this Roadmap (see **Table 1**) have rich potential that has more to give and it is our hope that in laying out the current state of affairs further progress will be catalysed adding to the remarkable achievements made so far.

Sir John B. Pendry
Imperial College London
2023

<table><caption>**Table 1.** The list of topics and authors in this roadmap.</caption><thead><tr><th>Section No.</th><th>Title</th><th>Authors</th></tr></thead><tbody><tr><td>1</td><td>Introduction to Roadmap</td><td>Vasily N. Astratov</td></tr><tr><td colspan="3"><em>Theme I: Information Science And Novel Structured Illumination</em></td></tr><tr><td>2</td><td>Deep Unrolled Image Restoration in Super-Resolution Microscopy</td><td>Yair Ben Sahel, Yonina C. Eldar</td></tr><tr><td>3</td><td>Deep Learning: Enabled Super-Resolution Microscopy</td><td>Luzhe Huang, Aydogan Ozcan</td></tr><tr><td>4</td><td>The Future of Super-Resolution Optical Microscopy at the Age of Big Data</td><td>Nikolay I. Zheludev</td></tr><tr><td>5</td><td>Label-Free Plasmonic/Metamaterial Structured Illumination Microscopy</td><td>Junxiang Zhao, Zachary Burns, Zhaowei Liu</td></tr><tr><td>6</td><td>Fundamental Limits to the Resolution of Label-free Far-field Microscopy</td><td>Evgenii Narimanov</td></tr><tr><td colspan="3"><em>Theme II: Interferometric And Holographic Approaches</em></td></tr><tr><td>7</td><td>Current Trends In Quantitative Phase Imaging</td><td>Neha Goswami, Gabriel Popescu</td></tr><tr><td>8</td><td>The Present and Future of Scattering Microscopy for Studying Biomolecular Mechanisms</td><td>Emanuel Pfitzner, Philipp Kukura</td></tr><tr><td>9</td><td>Coherent Brightfield Microscopy</td><td>Yi-Teng Hsiao, Chia-Lung Hsieh</td></tr><tr><td>10</td><td>Plasmon-Enhanced Optical Ptychography and Phase Imaging</td><td>Brian Abbey</td></tr><tr><td>11</td><td>Circular Intensity Differential Scattering Microscopy</td><td>Alberto Diaspro, Aymeric LeGratiet, Paolo Bianchini</td></tr><tr><td>12</td><td>Super-Resolved Off-Axis Holographic Multiplexing</td><td>Natan T. Shaked</td></tr><tr><td>13</td><td>Tomographic Diffractive Microscopy</td><td>Bertrand Simon, Nicolas Verrier, Matthieu Debailleul, Olivier Haeberlé</td></tr><tr><td>14</td><td>Scanning Near-Field Optical Microscope for Quantitative Nano-Imaging and Nano-Spectroscopy</td><td>Sheng Wang, Mengkun Liu</td></tr><tr><td colspan="3"><em>Theme III: Nonlinear Optics Approaches</em></td></tr><tr><td>15</td><td>Absorption-Based Super-Resolution Imaging</td><td>Yeran Bai, Ji-Xin Cheng</td></tr><tr><td>16</td><td>Pump-Probe Super-Resolution Microscopy</td><td>Paolo Bianchini, Behjat S. Kariman, Alberto Diaspro</td></tr><tr><td>17</td><td>Super-Resolution Raman Microscopy</td><td>Katsumasa Fujita</td></tr><tr><td>18</td><td>Silicon Non-Linearity Based Super-Resolved Pump-Probe Imaging</td><td>Moshe Sinvani, Zeev Zalevsky</td></tr><tr><td>19</td><td>The Rise of Mie-Tronics-Based Nonlinear Scattering Imaging</td><td>Xiangping Li, Guan-Jie Huang, Shi-Wei Chu</td></tr><tr><td>20</td><td>Label-Free Super-Resolution Microscopy by Non-Linear Photomodulated Reflectivity</td><td>Omer Tzang, Dror Hershkovitz, Ori Cheshnovsky</td></tr><tr><td>21</td><td>Structured Illumination-Based Nonlinear Label-Free Super-Resolution Microscopy</td><td>Mikko J. Huttunen, Stefan G. Stanciu</td></tr><tr><td colspan="3"><em>Theme IV: Advanced Superlens Designs</em></td></tr><tr><td>22</td><td>Surface Electromagnetic Wave Lenses and Transformation Optics</td><td>Vera N. Smolyaninova, Igor I. Smolyaninov</td></tr><tr><td>23</td><td>Perfect Imaging</td><td>Ulf Leonhardt, Sahar Sahebdivan</td></tr><tr><td>24</td><td>Microsphere Superlens and Metamaterial Solid Immersion Lens</td><td>Zengbo Wang, Boris Luk'yanchuk, Limin Wu</td></tr><tr><td>25</td><td>Microspherical Superlens Imaging: Fundamental Origins of Super-Resolution</td><td>Alexey V. Maslov, Vasily N. Astratov</td></tr><tr><td>26</td><td>Smartphone Ball Lens Microscopy</td><td>Vasily N. Astratov, Boya Jin</td></tr><tr><td>27</td><td>Imaging Beams for Subwavelength Resolution</td><td>Constantin R. Simovski</td></tr><tr><td>28</td><td>Microsphere-Assisted Interference Microscopy</td><td>Stephane Perrin, Paul Montgomery, Sylvain Lecler</td></tr></tbody></table>
# 1. Introduction to Roadmap

*(Vasily N. Astratov)*

Label-free imaging does not rely on the use of fluorescent markers and it has been a historically first and primary method of observational microbiology since the discovery of various microorganisms (“animalcules”) by Antonie van Leeuwenhoek in the 17th century. In this Roadmap, we address a development of label-free super-resolution (LFSR) imaging aimed at overcoming the classical diffraction limit introduced in the late 19th century by Abbe,[1] Helmholtz,[2] and Rayleigh.[3] According to the Abbe limit the far-field resolution of optical systems is limited to $\lambda/(2n \times \sin\theta) = \lambda/(2 \times NA)$, where $\lambda$ is the operating wavelength, $n$ is the refractive index of the surrounding medium, $\theta$ is the half-angle of the objective acceptance cone, $NA = n \times \sin\theta$ is the numerical aperture of the imaging system. This corresponds to about 0.2 µm in lateral direction and about 0.6 µm in the direction of the optical axis for visible light. The beginning of the 20th century was a period of extraordinary developments in physics, but at that time it might seem almost impossible that the diffraction limit would be overcome.

This introduction, however, would be incomplete without mentioning the remarkably successful super-resolved fluorescence (FL) microscopy based on staining biomedical samples with organic dyes (linked to labeling agents) or by tagging proteins of interest with fluorescent proteins. Due to its unrivaled contrast, FL imaging has emerged as one of the dominant light microscopy methods in modern biology. It allows one to “highlight” the structures inside a living cell and it became widespread since the discovery of green FL protein in the 1960s. More recent developments aimed at achieving FL-based far-field super-resolution can be divided into deterministic methods using certain nonlinear properties of such emitters and stochastic methods using their statistical single-molecule FL detection properties, while the distinction between these methods is also related to the way fluorescence is induced and read out. We refer the interested reader to a roadmap,[4] topical reviews,[10, 5-9] and books[11-13] in this area based on the following methods which all, in one way or another, address the problem of the limited spatial resolution in far-field microscopy: stimulated emission depletion (STED),[14, 15] ground state depletion (GSD),[16, 17] reversible saturated optical (fluorescence) transitions (RESOLFT),[18, 19] photoactivation localization microscopy (PALM),[20, 21] stochastic optical reconstruction microscopy (STORM),[22, 23] super-resolution optical fluctuation imaging (SOFI),[24, 25] structured illumination microscopy (SIM),[26-29] or saturated structured illumination microscopy (SSIM).[30] While SIM achieves a two-fold improvement in spatial resolution compared to conventional optical microscopy, STED, RESOLFT, PALM/STORM, or SSIM have all gone beyond this, pushing the limits of optical image resolution to the nanometer scale.[4] An additional development is represented by the 4Pi laser scanning FL microscope with an improved axial resolution.[31-33] The 2014 Nobel Prize in Chemistry awarded to Eric Betzig, Stefan W. Hell, and William E. Moerner “for the development of superresolved fluorescence microscopy” can be seen as a combined prize for single-molecule detection and superresolution FL imaging. While FL microscopy is a separate area from LFSR imaging, and it is generally beyond

the scope of this Roadmap, the fast progress in this area was inspirational for the recent developments in LFSR imaging.

In contrast to super-resolved FL microscopy, the development of LFSR area was a slow gradual process based on introducing novel physical mechanisms or interdisciplinary concepts symbolically represented in the tree diagram in **Figure 1** explained later in this Introduction. Since the label-free imaging does not rely on using FL markers, it promises practically unlimited applications spanning from visualization of subcellular structures to nanostructure characterization in materials science. However, label-free super-resolution (LFSR) microscopy has to rely on subtler light-scattering processes in nanoscale objects, resulting in lower image contrasts. The problem of overcoming the far-field diffraction limit is extremely challenging in view of the limited photon budgets offered by unstained nanoscale objects.

Our motivation to create the Roadmap on LFSR imaging stems from three major conceptual breakthroughs in this area which took place mainly in the last two decades. The first one is recognizing imaging as an information science problem based on using different forms of prior knowledge about objects, training of imaging systems considered as deep-learning networks, and, finally, using artificial intelligence for recognizing images and improving resolution of optical instruments. The second one is the development of structured illumination microscopy (SIM) where different forms of structured illumination such as interference patterns, superoscillatory optical beams, plasmonic hot spots, photonic nanojets, vertices, or hyperstructured beams were used to shift high spatial frequencies into the bandpass of the microscope. The third one is the appearance of novel concepts such as “perfect” lens, superlens, hyperlens, superoscillatory lens, and transformation optics in the context of development of nanoplasmonics and metamaterials. Each of these three areas has some history of development by the early pioneers in the 20th century, however they were transformed into new frontiers in optics and photonics research due to their explosive growth during the last two decades. Although these three advancements represent cornerstones of the ongoing revolution in LFSR imaging, they do not represent the entire LFSR field. There are other important approaches which have a long and distinguished history of development. They are based on near-field scanning, nonlinear optics, and interference-based detection techniques. *Thus, this Roadmap represents an attempt to combine under the same umbrella research efforts of many groups working on all these directions, create a comprehensive vision of the entire LFSR field, present current state-of-the-art in LFSR imaging, and predict future progress in this field.*

The history of development of LFSR imaging and genealogical connections between different areas are illustrated by a tree diagram in Figure 1. The stem of the tree diagram, indicated as “4 - Mainstream”, represents two types of advancements: i) major factors impacting microscopy such as invention of lasers as well as contributions of great pioneers in the field, and ii) development of interferometric and holographic approaches. The stem (4) of the tree represents the mainstream microscopy which became more powerful, advanced, and probably most widely used in today’s biomedical research. Although, technically, most of these methods have a diffraction-limited resolution, they really became a powerhouse for all other methods with the true super-resolution capability illustrated by the branches of the tree (1-3, 5-7), each branch representing a particular mechanism
<figure><img src="image_4.png" /><figcaption>**Figure 1.** Tree diagram of evolutionary development of LFSR subjects and methods. The tree is rooted in a classical diffraction limit introduced by Abbe, Helmholtz, and Rayleigh. The stem (4) represents development of *Mainstream* diffraction-limited microscopy due to incorporation of interferometric and holographic approaches. The branches represent different LFSR imaging mechanisms: 1 – *Near-field* scanning, 2 – *Information (INFO)* approach, 3 – *Nonlinear* optics approach, 5 – *Structured Illumination*, 6 – advanced *Superlens Designs*, 7 – *Microspherical* superlens imaging (MSI). *Super-Resolved FL Microscopy* is illustrated, but it does not belong to LFSR methods. Notation: NSOM, near-field scanning optical microscopy; s-NSOM, scattering-NSOM; EM, electromagnetic; SR, super-resolution; LSPARCOM, learned sparsity based super-resolution microscopy; DL-SR, deep learning-enabled image super-resolution; DSTM, deeply subwavelength topological microscopy; SRS, stimulated Raman scattering; CARS, coherent anti-Stokes Raman scattering; SHG, second harmonic generation; 2PEF, two-photon excitation fluorescence; NPMR, nonlinear photo-modulated reflectivity; SIM, structured illumination microscopy; DHM, digital holographic microscopy; CII, computational integral imaging; QPI, quantitative phase imaging; iSCAT, interferometric detection of scattering; COBRI, coherent brightfield imaging; CIDS, circular intensity differential scattering; DHM, digital holographic microscopy; TDM, tomographic diffractive microscopy; ROCS, rotating coherent scattering; Adv. SIM, advanced SIM; PSIM, plasmonic structured illumination microscopy; LPSIM, localized PSIM; MAIN, metamaterial-assisted illumination nanoscopy; 2DPN, 2-D plasmonic nanoscope; FFSL, far-field superlens; “Hyper”, short for “hyperlens”; FFTR, far-field time reversal; HIRES, high-index resolution enhancement by scattering; MFE, Maxwell fisheye; SOL, super-oscillatory lens; nSIL, nano solid immersion lense; mSIL, metamaterial SIL; MSI, microspherical superlens imaging; iMSI, interferometric MSI.</figcaption></figure>

or approach to the LFSR imaging: 1 – near-field scanning, 2 – information (INFO) approach, 3 – nonlinear optics approach, 5 – structured illumination, 6 – advanced superlens designs, and 7 – microspherical superlens imaging (MSI). The main methods of LFSR imaging with the key references are indicated on the tree diagram. More complete lists of publications devoted to different LFSR mechanisms can be found in the Topical Review articles.

We also illustrated super-resolved FL microscopy as an additional branch with the corresponding key references.[10–33, 4–9]

This tree diagram is a useful image for representing the entire LFSR field and its evolution in time, but it is rather difficult to follow the order of the tree diagram for creating the organizational structure of this Roadmap. Ideally, it could have been done by presenting Topical Review sections (called “Topics” for briefly)
either from bottom up or from left to right branches of the tree
diagram. Each Topic, however, focuses on the present and future
of a given field which means that the Topics tend to aggregate to-
wards the top of the tree diagram. In addition, different branches
of the tree are populated unevenly. As an example, the left-most
"near-field" branch (1) of the tree is represented by a single Topic
14. At the same time, each Topic is in some sense a mini-review
addressing several different developments inside each branch of
the tree and sometimes involving an interplay between different
branches. For these reasons, an attempt to closely follow the order
of the tree diagram is rather difficult to implement and, perhaps,
it would not be a natural approach in this situation.

To create an organizational structure for this Roadmap con-
taining an Introductory section and 27 Topical Review sections,
we used a somewhat simplified approach which still has a strong
relevance to the tree diagram in Figure 1. In this Roadmap,
the Topics are divided into four broad Themes. We begin this
Roadmap with Theme I devoted to the information science and
structured illumination directions (Topics 2–6) as areas at the
forefront of research in this field today. Theme I can be mapped to
combined second and fifth branches of the tree diagram. Theme
II combines more traditional interferometric and holographic ap-
proaches (Topics 7–14) corresponding to the stem (4) of the tree
diagram. Theme III describes the nonlinear optics approaches
(Topics 15–21) corresponding to the third branch of the tree. The
novel advanced superlens designs are combined in Theme IV
(Topics 22–28) which corresponds to combined sixth and seventh
branches of the tree diagram.

In this Introduction, we provide a brief account of history and
developments of the LFSR imaging field following the order of
branches (1–7) on the tree diagram with the references to the cor-
responding Topical Review sections.

**1.1. Near-Field Scanning**

The onset of the super-resolution era was marked by a proposal
of microscopy based on scanning of optical near fields in 1928
by Synge[34] who was encouraged by Albert Einstein. Much later,
in the 1980s and 1990s, this idea gave rise to near-field scan-
ning optical microscope (NSOM)[35, 36] and scattering-NSOM (s-
NSOM).[37, 38] Further development of these methods made pos-
sible electromagnetic field mapping in plasmonic and photonic
nanostructures.[39, 40] It also led to the development of the opti-
cal force microscopy.[41, 42] The timeline of these developments is
represented by the leftmost 1st branch on the Roadmap tree. Al-
though this approach was historically significant and providing
one of the highest, almost atomic-scale resolutions published to
date, its representation in this Roadmap is rather limited since
these methods are based on a relatively slow point-by-point near
field scanning by nanoapertures or nanoprobes. An account of
NSOM developments for quantitative nano-imaging and nano-
spectroscopy can be found in Topic 14 and in a review.[38]

**1.2. Information Approach**

In traditional optical imaging systems, the spatial resolution is
limited by the physics of diffraction, which acts as a low-pass filter

attenuating high spatial frequencies and retaining low frequen-
cies unchanged. In the information approach, however, the spa-
tial frequencies have to be transmitted by sampling as a result
of converting a signal into discrete numbers. This means that to
transmit the spatial "cut-off" frequency which determines the res-
olution, one has to transmit its coordinates at a series of points
spaced with the double frequency.[7] Imaging can be viewed as
an information communication problem where achieving higher
resolution requires higher channel capacity in a spirit of the in-
formation theory developed by Shannon in 1948.[43] Application
of the information approach to optics was prepared by the early
work of Toraldo di Francia who proposed in 1952 that the spa-
tial frequency cut-off of an imaging system can be extended.[44]
It was followed by introducing the mathematical framework for
the informational capacity of two-dimensional optical system,[45]
the formulation of invariance theorem by Lukosz,[46, 47] and in-
corporation of a noise factor in the theorem of invariance of the
information capacity of an optical image by Cox and Sheppard.[48]
The central concept is represented by the theorem of invariance
of information capacity which states that it is not the spatial
bandwidth but the information capacity of an imaging system
that is constant. The expression for the information capacity has
been derived based on three spatial dimensions, temporal dimen-
sion, and two independent states of polarization.[48] This means
that if the object belongs to a restricted class, it is in principle
possible to extend the spatial bandwidth (spatial resolution) by
encoding-decoding the additional information onto the indepen-
dent (unused) parameter(s) of the imaging system.[48] This early
work came to two revelations about the nature of super-resolution
which laid out the foundation for the future development in this
area. The first was that the invariance theorem shows that the
resolution can be enhanced by incorporating a priori knowledge
about the object. The second was that from a purely informational
point of view there are no bounds to achievable resolution, but the
practical bound is imposed by the signal-to-noise ratio (SNR) of
the resulting image.[48]

Meanwhile computational imaging has been developing
since the emergence of machine learning in the 1960s.
The most important advancements were the use of multi-
layered computational geometries such as deep neural and
backpropagation[49] networks as well as the development of com-
pressed sensing.[50, 51] The latter showed that with a prior knowl-
edge (assumption of sparsity), the signal may be reconstructed
with fewer samples than it is required by the Nyquist–Shannon
sampling theorem. It directly stimulated application of similar
principles to achieving super-resolution imaging.[52, 53]

In the past few years, the information approach sparked a
wealth of ideas where deep learning is directly applied to multi-
layer imaging networks for achieving super-resolution. Initially,
the applications were developed for FL microscopy. Topic 2
describes the development of FL single-molecule localization
microscopies such as PALM, STORM, and SPARCOM (sparsity-
based super-resolution correlation microscopy, which exploits
the sparse nature of the fluorophores distribution, alongside a
statistical prior of uncorrelated emissions) towards incorporating
algorithms with trained neural networks via a process known
as algorithmic unrolling that gave rise to Learned SPARCOM
(LSPARCOM).[54] The latter method enables efficient, accurate,
and robust imaging well below the diffraction limit, without any
prior knowledge of the imaging setup's physical parameters. It is proposed that algorithmic unrolling can become a powerful resource in label-free interferometric microscopy. Topic 3 is based on the first report on deep learning-enabled image super-resolution (DL-SR) in FL optical microscopy<sup>55,56</sup> and on applications to label-free microscopy such as digital staining of label-free high-resolution images.<sup>57</sup> Topic 4 is devoted to deeply subwavelength topological microscopy (DSTM) which is a truly label-free method based on using a deep learning neural network that has been trained on scattering events from a large set of a priori known objects.<sup>58</sup> This method provided measurements of the width of sub-wavelength slits on an opaque screen with an extraordinary accuracy of ≈λ/130 for a single-shot measurement or ≈λ/260 (i.e., 2.4 nm) when combining measurements of diffraction patterns at different distances from the object, thus challenging the accuracy of scanning electron microscopy and ion beam lithography.<sup>59</sup> Resolution analysis based on the Shannon's theory of information transmission in linear systems in the presence of noise<sup>60</sup> can be found in Topic 6. It develops earlier work<sup>48</sup> and incorporates both SNR and a priori knowledge about the object (in the form of sparsity) into the resolution limit determined by the information theory.

The conclusion of classical information theory that the noise factor determines the resolution of an imaging system is consistent with the Feynman's perspective that "Rayleigh's criterion is a rough idea in the first place", and a better resolution can be achieved "if sufficiently careful measurements of the exact intensity distribution over the diffracted image spot can be made".<sup>61</sup> It should be noted that the quantum physics approach based on quantum photon correlation allows increasing the resolution of images by the factor of two beyond the diffraction limit.<sup>62-64</sup> This approach considers anti-bunching photon statistics or quantum entanglement for photons sharing common location information. In principle, it can be interpreted as a form of using a priori knowledge about the sample or as a form of optical nonlinearity due to multiphoton nature of interaction with the sample, however, this approach is not included in this Roadmap.

### 1.3. Nonlinear Optics

The Abbe resolution limit is formulated for linear optical systems, however in special situations it can be surpassed even in this limit with the example given by the interference-based FL SIM where the two-fold resolution increase beyond the classical diffraction limit is still diffraction-limited.<sup>26-29</sup> The axial resolution can be enhanced severalfold compared to that of the standard FL confocal microscopy using a 4Pi microscope.<sup>31-33</sup> Optical nonlinearity opens new ways of achieving super-resolution imaging. The photoswitching and saturation (depletion) are nonlinear effects used in super-resolved FL microscopy. Another possibility is offered by the multiphoton microscopy, where the nonlinear processes provide novel, label-free contrast mechanism. Multiphoton microscopy can also increase the penetration depth by using longer excitation wavelengths and enhance resolution, contrast, and molecular specificity. It should be noted, however, that higher photoexcitation intensities intrinsic for nonlinear methods can limit their applications for studying biomedical samples. However, they can become a tool of choice for nanostructure

characterization in material science with many examples such as imaging single-walled carbon nanotubes, nanodiamonds, gold nanoparticles, graphene, silicon, etc. The development of nonlinear optical microscopy began with the broad application of lasers in the 1960s that resulted in the creation of stimulated Raman scattering (SRS), coherent anti-Stokes Raman scattering (CARS), second harmonic generation (SHG), two-photon excitation fluorescence (2PEF), and many other types of microscopy techniques.

This Roadmap contains several Topics where various optical nonlinearities are exploited to increase the resolution of optical microscopy beyond the classical diffraction limit using a label-free contrast mechanism. As shown in Topic 15, absorption-based imaging can be performed in two modalities (without using near-field probes): a) optical photothermal detection<sup>65</sup> which can provide resolution up to ≈λ/10 and b) ground-state depletion in transient absorption processes for imaging.<sup>66</sup> Pump-probe methods described in Topic 16 provide an experimental basis for many imaging modalities including transient absorption, SRS, CARS, SHG, 2PEF, etc. Nonlinear light-matter interactions in these methods involve the second χ<sup>(2)</sup> and third χ<sup>(3)</sup> order of susceptibilities. In particular, imaging on graphene with pulse powers about 7.0 MW cm<sup>-2</sup> allows achieving a resolution of the order of ≈λ/10.<sup>67</sup> Topic 17 is devoted to imaging based on Raman scattering that include coherent methods such as CARS, SRS, and surface enhanced Raman scattering (SERS). High-resolution coherent Raman imaging based on CARS and SRS have been demonstrated based on saturation excitation (SAX) microscopy.<sup>68</sup> Topic 18 considers super-resolution imaging based on material nonlinearity due to plasma dispersion nonlinearity in silicon.<sup>69</sup> Topic 19 develops a nonlinear Rayleigh scattering microscopy based on plasmonic or dielectric particles with resonant properties (Mie-resonators) which provide a novel super-resolution contrast mechanism.<sup>70</sup> Topic 20 introduces LFSR imaging based on nonlinear photo-modulated reflectivity (NPMR) which takes place due to the nonlinear changes in the reflectance of materials induced by an ultra-short pump pulse.<sup>71</sup> Topic 21 is devoted to a combination of SIM with the optical nonlinear detection,<sup>72</sup> where SIM illumination can be understood broadly similar to that in the pixel reassignment<sup>73</sup> or image scanning microscopies.

### 1.4. Mainstream Microscopy: Interferometric and Holographic Approaches

Despite all the successes of FL microscopy which became a major tool in cell biology, the mainstream label-free microscopy remained very attractive because it does not require cells (or other structures) to be stained, thus avoiding associated phototoxicity and photobleaching effects. Although being diffraction-limited, these methods (especially the modern interference detection techniques) offer an unsurpassed precision of axial resolution combined with high temporal resolution that explains the widespread and high popularity of these methods in biomedical imaging applications. In addition, the resolution of these label-free methods can be increased beyond the classical diffraction limit by combining them with near-field scanning, information-based or superlens-based approaches. The development of mainstream label-free methods has a distinguished
history in the 20th century and it is actively developing in the last two decades, as symbolically represented by stem (4) in the Roadmap tree diagram in Figure 1.

The first notable step was observation of gold particles with diameters less than 4 nm in solid material in 1902[74] and silver particles with estimated sizes 50–77 nm in a colloidal solution in 1905 by Richard Zsigmondy using his *ultramicroscope* (1925 Nobel Prize in Chemistry). This breakthrough result can be viewed as an early sign of emerging era of nanotechnology. The second step based on introduction of phase-contrast microscopy (PCM) in 1930s by Frits Zernike (1953 Nobel Prize in Physics)[75] made a profound impact on label-free imaging. It was followed by the development of a large family of interferometric detection techniques such as differential interference contrast (DIC) microscopy proposed in 1952 by Georges Nomarski,[76] interference reflection microscopy,[77] and Mirau correlation microscopy.[78] The third step began with the invention of holography by Dennis Gabor in 1948 that stimulated ideas about combining holography and microscopy aimed at highly sensitive measurements of the thickness and refractive index of biological samples. In the 1990s, increasing power of digital image sensors and computers facilitated development of digital holographic microscopy (DHM).[79] Application of DHM to 3D objects with complicated shapes allowed obtaining their extended focused images without any mechanical scanning or using special phase plates.[80] Another form of 3D volumetric object reconstruction is based on using computational integral imaging (CII) where sensing of the 3D scene is carried out by a microlens array.[81] More recently, quantitative phase imaging (QPI) provided the enhanced stability of imaging by locking of the incident field with the scattered light.[82] An interesting development of QPI for label-free, high-resolution discrimination of live stem cells is based on its combination with the total internal reflection microscopy.[83] The fourth step was developing methods of interferometric detection of scattering (iSCAT) from single nanoparticles,[84] viruses,[85] and proteins. One more important step was developing visible-light super-resolution microscopy where a specimen is stepped through a localized coherent “probe” wavefront, generating a series of diffraction patterns at the plane of a detector.[86, 87]

This Roadmap contains several Topics descending from the phase-contrast microscopy pioneered by Frits Zernike and showing the past, present, and future of the interferometric detection methods. Topic 7 presents current trends in QPI including imaging of strongly scattering structures, imaging with computational specificity, and improved resolution using deep learning. Topic 8 is devoted to iSCAT microscopy for studying biomolecular mechanisms. Topic 9 introduces coherent brightfield imaging (COBRI)[88] which can be considered as a transmission counterpart of iSCAT. Topic 10 describes the current status and future prospects of plasmon-enhanced optical pychography.[87] Topic 11 is devoted to circular intensity differential scattering (CIDS) method[89] originated in the 1980s as a differential measurement of the circular right and left polarized light scattered by a specimen outside the absorption bands that separates this method from the circular dichroism. Although the mechanism of this effect is different from that used in interferometric or holographic microscopy approaches, Topic 11 represents potential of polarization microscopy based on using nonconventional detection modalities. Topic 12 is devoted to off-axis

DHM where holographic multiplexing of up to six simultaneous wave-front channels can be reconstructed to an extended synthetic aperture and the super-resolved image.[90] Topic 13 introduces tomographic diffractive microscopy (TDM, also referred to as synthetic aperture microscopy, tomographic phase microscopy, phase nanoscopy, optical diffraction tomography, etc.; see Ref. [91] for a general introduction to these techniques) and its modern challenges. Topic 14 generally covers near-field scanning methods such as NSOM, however, the interferometric schemes that simultaneously retrieve the near-field optical signal amplitude and phase are also widely used in NSOM.

## 1.5. Structured Illumination

Structured illumination microscopy (SIM) is a widefield method in which a series of periodic illumination patterns are superimposed onto an object resulting in a twofold increase in resolution.[26–29] SIM was developed mainly in the context of FL imaging based on using multiple incoherent point-like sources emitting light at mutually random phase. This “loss of phase” in FL microscopy allows understanding image formation due to the superposition of the individual intensity distributions created by the point-like sources rather than their amplitudes. It also underlines the mathematical apparatus of SIM method based on the fact that product of functions (object and its illumination pattern) in real space leads to a convolution of these functions in the reciprocal space. Due to this property, the information about higher spatial frequencies (larger *k*-vectors) of the object can be projected into an escape cone of the microscope objective accessible for the far-field imaging. Thus, such “loss of phase” in FL imaging as opposed to elastic scattering “is the reason that fluorescence structured illumination microscopy is able to shift out-of-band information”[27] into the detection passband”.[92, 93] An increase of resolution beyond the factor of two requires one of the following: nonlinear sample response, prior knowledge about the sample and its properties, or near-field excitation or emission.[92] Nonlinear SIM can be realized using saturation effects in fluorophores,[30] which can lead to photobleaching and therefore limit the applications in biomedical imaging. Alternatively, some possibilities of further enhancing the SIM resolution without invoking any nonlinearity were proposed based on the proximity projection grating scheme.[94] It should be noted, however, that label-free imaging involves scattered light, and due to a coherent nature of scattering, the resolution advantage is limited.[92, 93] A twofold increase in lateral resolution can be achieved by oblique, coherent illumination. Averaging several high-contrast and high-resolution images in combination with near-field excitation lays the foundations of synthetic aperture microscopy[95, 96] and rotating coherent scattering (ROCS) microscopy.[97, 98] It should be noted that due to its high imaging speed, wide field-of-view, and low photodamage in FL biomedical applications there is currently a significant interest in exploring the resolution benefits of both FL[99] and label-free versions of SIM.[100–103] A common feature of these advanced SIM schemes is based on using a near-field excitation/illumination typically combined with the use of high-index substrates such as GaP.[103]

Another way of increasing the resolution is based on using periodically patterned nanoplasmonic or metamaterial sub-
strates which can create higher spatial frequency illumination patterns. As shown in Topic 5, plasmonic structured illumination microscopy (PSIM) shows 2.6-fold resolution advantage[104] and localized plasmonic structured illumination microscopy (LP-SIM) shows threefold resolution advantage with biomedical samples.[105, 106] PSIM and LPSIM use interference patterns from counter-propagating surface plasmon polaritons and localized surface plasmon waves, respectively. They originate in metallic nano-disc arrays to generate illuminations structured at the scale beyond the traditional diffraction-limit. Metamaterial-assisted illumination nanoscopy (MAIN) can achieve much higher resolution than the standard SIM methods due to a hyper-structured illumination theoretically proposed in[107] and experimentally studied in various hyperbolic metamaterials.[108–110] A 6-fold resolution improvement over the diffraction limit has been reported for hyper-structured illumination with label-free imaging, resolving features down to ≈80 nm.[110]

A different approach to ultra-sharp focusing of beams is represented by the optical superoscillation phenomenon reviewed in a recently published Roadmap on Supercollisions.[111] Mathematically, this phenomenon is rooted in a fact that a band-limited function can vary arbitrarily faster than its fastest Fourier component, over arbitrarily long intervals.[112] Experimentally, the subdiffraction hot spots produced by array of nanoholes illuminated with coherent light and by binary zone plates were observed and applied for super-resolution imaging.[113–116] Supercollaboratory “needles” extending for tens of wavelengths along the axis can be explored for high-density data storage,[117] light-induced magnetic holography,[118] planar diffractive lenses,[119] and countersurveillance metaoptics.[120]

## 1.6. Advanced Superlens Designs

The information on sub-wavelength features is carried by evanescent waves which exponentially decay in the out-of-plane direction, thus precluding reconstruction of an object in the far field with resolution better than the diffraction limit. The general principle of superlens designs is to preserve the information about high spatial frequencies encoded in the objects' near-fields. In 1970, Nassenstein suggested an ingenious scheme in which the object is illuminated with evanescent waves, and a magnified image is obtained using a holographic technique.[121] This work paved the way to future superlens designs. The whole area of superlens designs stemmed from the inspirational proposal of a “perfect lens” by Sir John Pendry in 2000[122] based on using negative index materials. Previously, Victor Veselago realized that if a material had negative values for both the electric and magnetic permittivities, then its index of refraction would also be negative.[123] This lens, however, works only when the object and its image are located in the subwavelength proximity to the lens which is different from imaging a remote object. In addition, such “near-sighted” lens does not have an ability to magnify the image. It was followed by several advanced superlens designs which had a significant impact on emerging nanoplasmonics and metamaterials communities: (i) 2D plasmonic nanoscope (2DPN) with ≈λ/8 resolution[124] using image magnification at much shorter plasmonic wavelengths compared to that in air, (ii) far-field superlens (FFSL) with ≈λ/6 resolution[125] which en-

hances the evanescent waves and converts them into propagating waves, (iii) hyperlens[126, 127] represented by an anisotropic metamaterial with hyperbolic dispersion which can provide magnification in a cylindrical geometry, (iv) design based on the far-field time reversal (FFTR) for objects with near-field scatterers providing imaging with an up to ≈λ/30 resolution,[128] v) high-index resolution enhancement by scattering (HIRES) lens[129] with ≈λ/6 resolution in a visible regime, vi) broadband Maxwell fisheye (MFE) and inverted Eaton microlenses[130, 131] where the precondition for superresolution is a drain placed at the image position, vii) super-oscillatory lens (SOL) with ≈λ/6 resolution in a visible regime[111–120, 132] represented by a nanostructured mask which focuses coherent beam into a subwavelength spot, and viii) adiabatic lens[133] which utilizes the interference of slowly decompressing far-field electromagnetic waves in plasmonic waveguides with gradient properties.

Topic 22 presents a review of surface electromagnetic wave lenses and transformation optics.[124–131] It also introduces a newly discovered class of surface electromagnetic waves which propagate along gradual interfaces of lossy optical media. Topic 23 is devoted to a “perfect” imaging by MFE and Eaton lenses[130, 131] as well as, more generally, to transformation optics. A particularly interesting observation is that curved surfaces like the surface of the sphere modify the resolution of imaging in a fundamental way due to modification of the uncertainty relation between position and momentum.

## 1.7. Microspherical Superlens Imaging (MSI)

Magnification of near-field image components has been suggested in experiments with self-assembled plano-spherical nano solid immersion lenses (nSILs).[134, 135] Due to micron-scale dimensions of nSILs, the resolution was found to be slightly improved compared to macroscopic SILs limit ($\lambda/(2n)$), but it was still close to ≈λ/3.[135] In this regard, the observation of deeply subwavelength resolution by virtual imaging through microspheres in 2011[136] was a breakthrough result which stimulated a significant and prolonged interest of imaging community in this technology. To enable study of biomedical samples, MSI technology was developed by using liquid-immersed high-index microspheres.[137] Convolution-based resolution analysis revealed that the MSI resolution is typically limited at ≈λ/6-λ/7 level.[138] Exact numerical solutions of Maxwell equations, however, revealed that the maximal theoretical resolution is limited at ≈λ/4 level.[139] It stimulated a deeper theoretical investigation of the resolution criteria and underlying MSI mechanisms.[140–142] Because of the limited field-of-view through individual microspheres, several MSI scanning schemes were proposed including use of movable coverslips with embedded microspheres,[138] integration with AFM system[143] and several other methods. A notable advancement in the field was the development of metamaterial SILs (mSILs) with the material of the lenses formed by densely packed nanoscale TiO₂ clusters[144] which are capable of potentially even higher resolution compared to conventional MSI techniques.

Topic 24 represents a history of development and state-of-the-art of MSI and mSIL methods. Topic 25 contains a review of theoretical factors which can be responsible for super-resolution
capability of MSI including the fundamental role of curvature of microsphere, resonant properties, and role of plasmonic contributions. Topic 26 describes designs of pocket-size smartphone-based microscopes aimed at maximizing their resolution capability. Topic 27 represents a theoretical analysis of resolving power of MSI method inspired by analogy with hyperlens imaging mechanisms where the super-resolution is achieved without involving evanescent fields. Topic 28 contains an account of integration of MSI technology with interferometry (iMSI) which can be realized in Linnik,[145] Mirau,[146, 147] and Mach-Zehnder[148] configurations to achieve surface topography reconstruction with a high axial sensitivity.

In conclusion, this Roadmap provides a snapshot of this rapidly evolving field ultimately intending to develop a systematic approach to this area based on classification of physical mechanisms of LFRS imaging and to create a vision for current and future developments in this field.

This Roadmap originated from a series of special sessions and workshops on Label-Free Super-Resolution Microscopy, which I organized at PQE-2016 and 2017, IEEE Photonics-2017, and ICTON-2017–2020 conferences. Many of the topics included in this Roadmap are represented by Chapters in a book “Label-Free Super-Resolution Microscopy” (Ed. Vasily Astratov) published in 2019.[149] For these Topics, the interested readers can potentially find more detailed information on the corresponding subjects in this book.

# Theme I: Information Science and Novel Structured Illumination

## 2. Deep Unrolled Image Restoration in Super-Resolution Microscopy

*(Yair Ben Sahel*, Yonina C. Eldar)*

### 2.1. Introduction

Inspection of sub-cellular features and organelles requires imaging tools with nanometer resolution. Until the development of super-resolution microscopy, the optical resolution limit of was set by Abbe's diffraction limit, which is hundreds of nanometers at best for modern light microscopes. When dealing with labeled samples, like in fluorescence microscopy, it is possible to surpass the diffraction limit by distinguishing between light coming from two neighboring fluorophores.[150] Such distinction can be done by utilizing photo-activated or photo-switching fluorophores to separate light emission in time; this is the basic principle of single molecule localization microscopy (SMLM) techniques such as photo-activated localization microscopy (PALM) and stochastic optical reconstruction microscopy (STORM).[20, 22] These methods take a sequence of diffraction-limited images, generated by a sparse set of fluorophores with whose overlapping point-spread functions (PSFs) have minimal overlap, and produce emitter localizations with high precision. However, low emitter density requires lengthy imaging times to achieve full coverage and minimal overlap between PSFs, resulting in low temporal resolution.

To overcome the long acquisition times required for SMLM methods, a variety of techniques have emerged, which reduce

the number of frames required for reconstruction of the 2D super-resolved image.[151–153, 24] These techniques exploit prior knowledge regarding either the optical system, the geometrical structure of the sample, or the statistics of the emitters. One such technique is SPARCOM,[153, 154] which makes use of sparsity in the correlation domain, while assuming that the blinking emitters are uncorrelated over time and space. This allows to re-formulate the localization task as a sparse recovery problem which can be solved, for example, using the “Iterative Shrinkage and Thresholding Algorithm” (ISTA).[155] SPARCOM achieves similar spatial resolution compared to a standard STORM reconstruction with as few as 361 and even 60 frames, compared with the 12 000 frames for standard STORM recovery, corresponding to a 33- or 200-times faster acquisition rate. Thus, SPARCOM improves temporal resolution while maintaining the spatial resolution of PALM/STORM. However, these benefits also introduce some tradeoffs: SPARCOM requires prior knowledge of the PSF of the optical imaging system, which is not always available, and a careful choice of a regularization factor $\lambda$, which is generally done heuristically.

These disadvantages can be overcome by replacing iterative algorithms like SPARCOM with trained neural networks, where each block in the network corresponds to single iteration of the algorithm. This allows us to automatically learn parameters which would have to be specified explicitly or tuned, and relevant context that may have been ignored by the algorithm can be incorporated into the learned model. This process is known as algorithmic unrolling[156] (or alternatively, unfolding), and it enables model-based learning where we benefit from both the underlying structure and prior physical information, but at the same time learn from the data itself. Since its introduction a decade ago, a wide variety of techniques have been adapted using learned unrolling, enabling improvements in performance across a variety of settings.[157] Here, we briefly review how unrolling has been used to achieve fast, accurate super-resolution imaging in SMLM while training from even only a single image sequence and preserving interpretability. We then turn to discuss how algorithmic unrolling may be applied to data analysis problems in label-free microscopy, to achieve similar benefits.

### 2.2. Unrolling in Single Molecule Localization Microscopy

The shortcomings of iterative algorithms like SPARCOM can be overcome by learning from data using an algorithm unrolling approach. This was recently demonstrated by Dardikman-Yoffe et al.,[54] which introduced Learned SPARCOM (LSPARCOM) - a deep network with 10 layers resulting from unrolling SPARCOM, trained on a single sequence of frames taken from one field-of-view (FOV) with a known underlying structure which was generated via simulation. LSPARCOM is derived by unrolling ISTA into a learned ISTA (LISTA) network,[156] yielding an efficient and interpretable neural network which does not require fine-tuning or prior knowledge of the regularization factor of the system's PSF.

The results shown in **Figures 2 and 3** illustrate that LSPARCOM outperforms its iterative counterpart, and the leading classical deep-learning based localization method for high emitter-density frames, Deep-STORM.[160] LSPARCOM is also
<figure><img src="image_5.png" /><figcaption>**Figure 2.** Super-resolved reconstruction of a simulated tubulins dataset,[158] composed of 361 high-density frames. a,b) SPARCOM reconstruction, executed over 100 iterations with a) unknown PSF (assuming a dirac delta PSF) and $\lambda = 0.0105$, b) the correct PSF and $\lambda = 0.13$. c) LSPARCOM reconstruction. d) Deep-STORM reconstruction. e) Ground truth position. Reproduced with permission[54] Copyright 2020, Optical Society of America.</figcaption></figure>

faster to use, with approximately 5x improvement over SPARCOM in execution time. On the other hand, LSPARCOM is yet to provide us with fully accurate reconstructions. Its results tend to be somewhat fragmented, as can be seen by the missing connections between the artificial filaments in Figure 2d. Such impairments may be resolved by adjusting the unrolling scheme of LSPARCOM and expanding its training dataset to more than a single field-of-view, as suggested in.[54] Nonetheless, the improvement brought by unrolling in various imaging modalities,[161] and particularly by LSPARCOM in the context of SMLM, further suggests that unrolling may benefit other super-resolution imaging schemes, as we discuss in the next section.

## 2.3. Possible Extensions to Label-Free Microscopy

Learned unrolling has shown to achieve fast, highly accurate, interpretable results in SMLM. Thus, it is interesting to explore applications to other challenges in computational microscopy, and label-free microscopy in particular. Here we touch on two methods of label-free microscopy that may benefit from unrolling in particular: differential interference contrast (DIC) microscopy, and phase contrast microscopy. The image reconstruction task in each of these methods can be described using a linear image formation model[162, 163]:

$$g = Hf \qquad (1)$$

with g being the observed image, and H representing the image formation process of the microscope (i.e., the microscope's PSF). The ideal object image that we want to retrieve - be it the optical

path length distribution in the object, or artifact-free phase contrast image - is denoted by $f$. This model allows for re-formulation of the restoration task as a regularized quadratic optimization problem,[162, 163] that can be iteratively solved using non-negative multiplicative updating[164]:

$$\hat{f} = \min_{f \ge 0} \|g - Hf\|_2^2 + \gamma \|Rf\|_2^2 + \beta \|Wf\|_1 \qquad (2)$$

where R is a Laplacian matrix defining the similarity between spatial pixel neighbors, and W is a diagonal matrix which defines the sparsity term. Their relative importance is controlled by the positive coefficients $\gamma$ and $\beta$. The corresponding iterative algorithm for this problem is detailed in Algorithm 1 (full derivation of the algorithm is provided in[162]).

**Algorithm 1 Iterative Multiplicative Updating**

**Input:** $Q, z, \alpha, \beta, \epsilon$

**Output:** $\hat{f}$

1: $\hat{f}^{(1)} = 1, w^{(1)} = 1, k = 1$.

2: while $\|\hat{f}^{(k+1)} - \hat{f}^{(k)}\|_2^2 > \epsilon$ do

3: $\hat{f}_i^{(k+1)} = \frac{-(z_i + \beta w_i^{(k)}) + \sqrt{(z_i + \beta w_i^{(k)})^2 + 4(Q + \hat{f}^{(k)})_i (Q - \hat{f}^{(k)})_i}{2(Q + \hat{f}^{(k)})_i} \hat{f}_i^{(k)}$

4: $w_i^{(k+1)} = \frac{1}{\hat{f}_i^{(k+1)} + \alpha}$

5: $k \leftarrow k + 1$

6: end while

7: $\hat{f} = \hat{f}^{(k)}$

<figure><img src="image_6.png" /><figcaption>**Figure 3.** Super-resolved reconstruction of an experimental tubulins dataset,[158] composed of 15 000 low-density frames. The frames were summed in groups of 50, resulting in a high-density sequence of 300 frames, on which the super-resolved reconstructions were performed. a,b) SPARCOM reconstruction, executed over 100 iterations with a) unknown PSF (assuming a dirac delta PSF) and $\lambda = 0.0025$, b) the correct PSF and $\lambda = 0.1$. c) LSPARCOM reconstruction. d) Deep-STORM reconstruction. e) ThunderSTORM[159] reconstruction, using the original low-density sequence, which we refer to “ground truth” (there is no actual ground truth since this is an experimental dataset). Reproduced with permission[54] Copyright 2020, Optical Society of America.</figcaption></figure>
where $\alpha$ and $\epsilon$ are positive constants, $Q = H^T H + \gamma R^T R$, $z = -H^T g$ and $w = \text{diag}(W)$. Note that as part of the algorithm, we express $Q$ in terms of its positive and negative components:

$$Q_{i,j}^{+} = \begin{cases} Q_{i,j} & \text{if } Q_{i,j} > 0 \\ 0 & \text{otherwise} \end{cases}, \quad Q_{i,j}^{-} = \begin{cases} |Q_{i,j}| & \text{if } Q_{i,j} < 0 \\ 0 & \text{otherwise} \end{cases} \quad (3)$$

We can see that this method requires careful choice of several hyper-parameters ($\alpha, \beta, \gamma, \epsilon$), which can be highly time-consuming. In addition, estimation of the microscope's PSF (H) and the Laplacian matrix (R) are also required. This is where algorithm unrolling can come into play - by unfolding the iterative updates in Algorithm 1, a deep network can be formed. Instead of iterative computation with a stopping criteria, we will have a neural network with a set number of layers, L. Each layer will consist of trainable parameters $\{Q^{(l)}, z^{(l)}, \beta^{(l)}, \alpha^{(l)}\}_{l=1,...,L}$ and perform the same update rules as in Algorithm 1 (lines 3,4) - only with learned scalars and matrices instead of predefined ones. The specifics of the network's architecture and training (exact number of layers, tied/untied weights across different layers and loss function) need to be chosen to complete the solution. Nonetheless, given the improvement brought by algorithm unrolling in the case of super-resolution microscopy, we believe that such an extension will result in an efficient and interpretable network that may outperform its iterative counterparts for image reconstruction in label-free microscopy.

## 2.4. Conclusion

Algorithm Unrolling is a powerful model-based approach to develop deep neural networks, based on iterative algorithms that are used widely in signal and image processing. We have described how unrolling allowed to combine SPARCOM, a recent high-performing classical method for SMLM image restoration, with model-based deep learning, resulting in LSPARCOM, a compact and interpretable neural network that outperforms its iterative counterpart and other learning-based methods. Given the success of applying algorithm unrolling in super-resolution, we propose to extend its application to label-free imaging schemes that can also benefit from it, such as DIC and phase contrast microscopy, and provided a general outline for such an application.

# 3. Deep Learning-Enabled Super-Resolution Microscopy

(Luzhe Huang, Aydogan Ozcan*)

## 3.1. Introduction

Super-resolution (SR) has long been a hot topic in image processing field focusing on the reconstruction of a high resolution (HR) image from one or more low resolution (LR) image(s). Super-resolution imaging techniques have been widely used in various fields, including e.g., photography,[165] medical imaging[166] and microscopy.[167] For example, optical microscopy modalities, including bright-field and fluorescence microscopes,

are in general diffraction limited, which prevents standard microscopy systems to resolve objects separated by a distance that is closer than approximately half a wavelength, in other words, ≈200–300 nm for visible light. In order to overcome the diffraction limit, the last two decades have seen various advances in SR fluorescence microscopy techniques, e.g., stimulated emission depletion (STED)[14] microscopy, stochastic optical reconstruction microscopy (STORM),[22] photo-activated localization microscopy (PALM),[20] and structured illumination microscopy (SIM),[26] among others. These methods achieve SR following two basic principles: (1) by using illumination patterns that modulate the emission of fluorescent molecules such that closely-spaced molecules can be differentiated spatially, or (2) by modulating the temporal behavior of fluorescent molecules to resolve different molecules as a function of time. Through these advances, SR fluorescence microscopy has revolutionized our understanding of biology, including e.g., cellular and molecular structures, stoichiometric characterization of molecular complexes, and temporal dynamics within cells.[167]

In addition to these, deep learning-based computational methods have also enabled an unprecedented success in advancing microscopic imaging, with various new image reconstruction and enhancement tools. As illustrated in Figure 4a, one of the most widely explored applications of deep learning in optical microscopy has been image super-resolution. The first public report on deep learning-enabled image super-resolution in optical microscopy was introduced in May 2017,[168, 55] which also demonstrated an extended depth-of-field. Since then deep learning super-resolution (DL-SR) microscopy has been applied to various imaging modalities, including bright-field[169, 55] and wide-field fluorescence microscopy.[170–175] In addition to SR, deep learning has also been utilized for e.g., denoising low-quality microscopic images,[169, 171, 175] virtually refocusing microscopic images, on demand, to designated 3D surfaces,[176, 177] virtual staining of label-free microscopic images,[178, 179, 57] virtual re-staining of already-labeled microscopic images,[180] or prediction of microscopic fluorescent labels in label-free images.[181, 182]

In this section, we will focus on DL-SR microscopy and will first discuss the forward imaging model of optical microscopy in order to provide an introduction to some of the commonly employed deep neural network (DNN) architectures, also covering their applications in DL-SR microscopy. Finally, we will conclude with a discussion of some of the limitations of these emerging techniques, also exploring potential future directions of research.

## 3.2. Building Blocks of Deep Learning-Enabled SR Microscopy

Assuming $f$ denotes the object function, the forward optical imaging process can be modeled by the following discrete model[56]

$$g = P(Hf) + \epsilon \quad (4)$$

where $g$ is the vectorized optical intensity collected on the image sensor plane of the microscope. $P$ represents a Poisson random process, modelling the shot noise and $\epsilon$ stands for other noise terms in the system, including thermal noise and readout noise,
<figure><img src="image_7.png" /><figcaption>**Figure 4.** DL-SR microscopy and commonly employed DNN architectures. a) DL-SR microscopy digitally transforms a LR image obtained by a diffraction-limited microscope to match the corresponding HR image of the same specimen that is acquired by a SR microscopy modality.[172] b) Common DNN architectures frequently used for DL-SR microscopy, including encoder-decoder, U-Net and DenseNet.</figcaption></figure>

which can be in general modeled as additive white Gaussian noise. For fluorescence microscopic imaging, as an example, the transfer function $H$ can be generally modeled as a circulant matrix of the point spread function (PSF), which is determined by e.g., the numerical aperture of the objective lens of the microscope, the wavelength of light and potential aberrations in the imaging system. Super-resolution techniques focus on solving an inverse problem to retrieve the HR image $f$ from LR measurement(s) set by $g$. However, due to the low-pass characteristics of the transfer function and the existence of measurement noise and aberrations, this problem is underdetermined and ill-posed.

Some of the earlier generations of optical super-resolution microscopy methods rely on special illumination patterns or controlling the temporal behavior of fluorophores in order to separate emitters in space or time, such that $f$ becomes sparse and the inverse problem can be solved with adequate regularization and prior knowledge of $H$. However, limited by the space-bandwidth-product (SBP) of the microscopy system, multiple image acquisition steps are generally required for these SR microscopy methods, which prolong the imaging time and considerably increase total sample irradiation, which might introduce photo-bleaching and/or photo-toxicity.

Since the success of LeNet-5,[183] a 7-level convolutional neural network (CNN), in handwritten digit recognition in 1998, CNNs have gained massive popularity in various computer vision tasks

and have over time surpassed human experts in certain tasks. Convolutional layers share the same convolution kernels and biases such that the number of trainable parameters is significantly reduced compared to fully-connected layers. Another advantage of CNNs is that they permit variable-size input images, which is essential for processing sample images of different sizes. Two major types of CNN-based network architectures used in microscopic image super-resolution are U-Net[184] and DenseNet[185] (see Figure 4b). U-Net introduces skip-connections between the encoder-decoder layers to facilitate mapping of low-level and structural image features.[184] Although it was originally aimed at biomedical image segmentation, U-Net and its variants have been frequently used in various image transformation and reconstruction tasks in optical microscopic imaging, including super-resolution microscopy. As an example, Wang *et al*. employed a U-Net based architecture to realize cross-modality SR image transformations using diffraction-limited confocal images as input to match, at its output, the high-resolution images obtained by STED microscopy[172]; a similar success was also demonstrated by establishing a SR image transformation between diffraction-limited TIRF (total internal reflection fluorescence microscopy) and SIM, enabling the reconstruction of SIM equivalent images of live cells and embryos without the need for structured illumination, potentially reducing photo-toxicity and photo-bleaching on the samples.[172] Similar deep neural network architectures were also demonstrated to improve image resolution in wide-
<figure><img src="image_8.png" /><figcaption>**Figure 5.** Applications of DL-SR optical microscopy. a) STORM,[160] SIM,[191] PALM,[187] and Fourier ptychographic microscopy (FPM).[194] b) Single-image super-resolution in wide-field fluorescence,[172, 174] bright-field,[55] and digital holographic microscopy.[195] Reproduced with permission.[55, 160] Copyright 2017 and 2018, The Optical Society.</figcaption></figure>

field microscopy,[171, 172] STORM,[160] light-sheet microscopy[186] and PALM.[187]

Following the observation[188] that shorter connections between layers substantially improve the training of deeper CNNs, DenseNet[185] was constructed to connect every convolutional layer in each block with its preceding feature maps to alleviate diminishing gradients inside the network and strengthen the propagation and reuse of features within the deep network. The success of DenseNet in super-resolution optical microscopy has been demonstrated through transforming LR images obtained by a low numerical aperture (NA) objective lens, to match HR counterparts captured with a high-NA lens[173]; similarly, cross-modality SR imaging was also demonstrated using DenseNet, transforming diffraction-limited microscopy images to match HR images obtained by e.g., STED, SIM.[174, 175]

These earlier studies have mainly employed DL to solve the inverse imaging problem in two ways: (1) to enhance existing optical super-resolution microscopy modalities, and (2) to perform single image super-resolution (SISR) using a diffraction-limited microscope. As summarized in **Figure 5a**, DL has been successfully applied to advance various optical super-resolution microscopy modalities, including e.g., STORM,[160, 189, 190] PALM,[187] SIM,[191] FPM,[192–194] helping to reduce the total image acquisition time, accelerating the SR imaging speed as well as reducing photobleaching and

phototoxicity during the imaging process. For SISR, on the other hand, DNNs have been successfully demonstrated to transform a given LR image captured by a low-NA objective lens to match the ground truth labels generated by a high-NA objective lens,[172, 55] or to perform cross-modality image transformations by super-resolving a diffraction-limited microscopic image[172, 174] as shown in Figure 5b. Deep learning-enabled SISR and related techniques have been widely successful to improve the resolution of e.g., wide-field fluorescence microscopy,[172, 174] bright-field microscopy,[169, 55] digital holographic microscopy,[195] among others.

### 3.3. Conclusions and Future Prospects

Fueled by the rapid advances in DL, the massive availability of high-end computational hardware and GPUs, as well as the precision, repeatability, and high-speed of microscopic imaging systems have opened up a new set of exciting opportunities that are data-driven in solving inverse problems in optical microscopy through learning. Nevertheless, for wide-scale adoption of DL-SR microscopy techniques in e.g., biomedical and healthcare-related tasks, model generalizability, interpretability, and reliability are still open research questions for DNN models due to their “black-box” behavior stemming from the complexity of such
deep neural networks with millions of parameters. Modern neural networks have been shown to generalize well as long as the testing data distribution is close to that of the training data. Failure modes of DNNs when a distribution shift occurs in the testing phase need comprehensive investigation, demanding the development of new statistical tools that will routinely warn the users, as a watchdog, when such changes/drifts start to emerge. Model interpretability and reliability are especially critical for clinical, biomedical applications, where imprecise and unexplainable predictions can potentially lead to severe negative consequences. Further interpretation and better understanding of the underlying image features and the network models' statistical inference process may not only improve the reliability of DNN models, but also provide new insights regarding the optical forward model and the physics of the microscopic imaging process that are hard to model with our current understanding of light-matter interaction. An important example of this includes the recently demonstrated cross-modality image transformations from holography to brightfield microscopy.[196] reconstructing object features that are holographically imaged using a single wavelength, spatially-coherent illumination into brightfield equivalent images, with the color and spatial contrast of a bright-field microscope that uses broadband and spatially-incoherent light.

In addition to the choice of the network architecture, the training dataset quality and the image registration processes are critical factors influencing the network performance. Careful registration and matching between the input and ground truth data are generally required to produce high-quality training image datasets, and this can be realized by e.g., digitally generating training input images from ground truth images, or through rigid/non-rigid transformation-based image registration processes. As another very important factor that determines the success of a DNN-based SR model, the training loss function often consists of structural similarity metrics between the network's output and the ground truth images, including e.g., pixel-wise difference loss terms such as mean absolute error (L1-norm), mean squared error (L2-norm), BerHu loss,[197] and structural similarity index (SSIM) loss, among others. In addition to these loss terms, Generative Adversarial Network (GAN)[198] framework also provides another type of training loss (the discriminator loss) by optimizing an evaluation network, which adaptively helps the training of the generator network to produce accurate super-resolved and enhanced images. Other types of loss functions including perceptual loss[199] that are derived from pre-trained network models are also shown to be effective in microscopic image enhancement and SR tasks.

In the future, we believe that deep learning-based methods will play an increasingly important role in advancing optical microscopy techniques, and will find various practical applications, especially in areas where automation and high-throughput image analysis are required. Ultimately, the use of deep learning in super-resolution microscopy could create fundamentally new generations of optical instruments and measurement systems, designed by joint optimization of the optical front-end hardware (e.g., through trainable diffractive processors[200, 201]) and the back-end image processing workflow, achieving high-resolution imaging with low-cost and/or high-throughput microscopy systems.

# 4. The Future of Super-Resolution Optical Microscopy at the Age of Big Data

(Nikolay I. Zheludev*)

We are at the verge of a new era of microscopy, the era of Deeply Subwavelength Optical Microscopy (Figure 6). This technology is based on the recovery of information on the shape and size of the object from its light scattering pattern using neural network-based artificial intelligence. Adding topologically structured illumination improves resolution that reaches molecular and even atomic levels. Imaging of soft-matter biological and solid-state nanotechnology samples will be equally possible in color and in two and three dimensions without labeling samples with luminescent material and at low intensities, without using nonlinear light-matter interactions. Training of the neural networks is a “Big Data” scale problem, but at the user end the optical part of the microscopy instrument is simple and only modest computer resources are required.

The label-free imaging is an ill-posed inverse light scattering problem, which can be reduced to solving the Fredholm integral equation.[204] Not so long ago, my close colleague Prof. Ilya Kuprov gave me a call saying: “Nikolay, I now know why your artificial intelligence-based high-resolution imaging techniques work so well. I found a paper[205] mathematically proving that the Fredholm integral equation can be accurately solved by a fully connected neural network!” This was exciting and very important news. We have been enthusiastically developing Deeply Subwavelength Optical Microscopy enabled by neural network-based artificial intelligence that can see far beyond the diffraction limit.[206–208, 59]

In conventional microscopy, the image of the object is constructed by a lens. Such imaging has a “diffraction limit” on resolution, which depends on the numerical aperture of the lens, but cannot exceed half of the wavelength of light. With artificial intelligence we beat the diffraction limit several orders of magnitude over by recording the intensity pattern of coherent light scattered from the sample into the far-field. In fact, we detect only the intensity of scattered light by an image sensor. This information is not sufficient to exactly reconstruct shape of the scattering object by analytical techniques. However, we retrieve information about the unknown object using a deep learning neural network that has been trained on scattering events from a large set of a priori known objects. Importantly, the network does not operate as correlator with a library of a priori known objects, but as a true solver of the inverse scattering problem. For instance, with a laser operating at 633nm, we retrieve the width of a randomly positioned sub-wavelength slits in opaque screen with accuracy of ≈λ/130 for a single-shot measurement or ≈λ/260 (i.e. 2.4nm), when combining measurements of diffraction patterns at different distances from the object. Such accuracy challenges the accuracy of scanning electron microscopy and ion beam lithography that were used to manufacture and measure the slits. In numerical experiments, we show that the technique could reach an accuracy beyond λ/1000, which brings it to the atomic scale.[59]

The use of artificial intelligence requires revising of what imaging and microscopy are in the age of big data. For centuries, imaging was a technique of representation of an object's shape by creating a light pattern resembling the object, in the way that a
<figure><img src="image_9.png" /><figcaption>**Figure 6.** Deeply Subwavelength Topological Metrology and Microscopy with a topologically structured (e.g., superoscillatory) light field (following[202, 203]). The intensity profile of the diffraction pattern resulting from scattering of the topologically structured light field on the imaged object is mapped by the image intensity sensor. The artificial intelligence deconvolution program, trained on a large number of scattering events with a priori known object (numbered 1,2,3,4 ...) reconstructs/measures the object from the collected data with high resolution.</figcaption></figure>

microscope creates it on the retina of the observer's eye, or pro-
jector at the cinema screen. Data processing techniques imple-
mented on computers can enhance or completely replace the lens
as the handler of optical information. Moreover, image analysis
and mining data from the image do not necessarily involve pro-
jecting them on the screen as they can remain in the computer
memory. Indeed, computer-enabled imaging is commonly used
far- and near-field optical microscopies and the electron-beam
imaging techniques. The artificial intelligence generated images
also fall into this category. Broadly, imaging is the representation
of an object's form and involves the generation, collection, du-
plication, analysis, modification, and visualization of images, in-
cluding imaging things that the human eye cannot detect. The
artificial intelligence-enabled imaging technique provide repre-
sentations of the object's including its form, shape and dimen-
sions and are true imaging techniques. As we have shown, they
can be extremely useful in microscopy "to view objects ... that
cannot be seen with the naked eye (objects that are not within
the resolution range of the normal eye)."[209]

Reconstruction of the object shape from its far-field diffrac-
tion pattern shows that a lens is not the only tool for construct-
ing the image. Moreover, the image may be constructed much
more precisely than the lens can do. What does it makes the
improved resolution possible? From the prospective of the infor-
mation theory, multiple scattering patterns recorded during the
training process provide more information for the image retrieval
process than what is available in a lens-generated single image.
Moreover, for some objects, the object's sparsity and the exis-
tence of prior knowledge about it significantly helps the retrieval
process.

How can we best exploit the ability of artificial intelligence to
handle complex stuttering patterns and to achieve higher resolu-
tion of optical imaging instruments?

We recently observed that the scattering of topologically struc-
tured light could be very sensitive to the placement and dis-
placement in the field of an absorbing nanoparticle with a
size of only λ/1000 (about a covalent diameter of the caesium
atom).[207] Complex monochromatic light fields are known to
have deeply subwavelength structures, such as phase singular-
ities, zones of energy backflow and rapid phase variation and en-
ergy hotspots.[210] This has already been used in metrology.[211]
With such topologically structured illumination scattered field
varies substantially when deeply subwavelength features of the
light field overlaps with nanoscale features of the scatter. The
reaction of topologically structured light field to the present
of deeply subwavelength object could be two to three orders
stronger compared to the plane wave illumination.

These works have resulted in developing the Deeply Subwave-
length Topological Microscopy.[202, 207] It reveals the fine structure
of an object through its far-field scattering pattern when it is il-
luminated with light containing deeply subwavelength singular-
ity features. The object is reconstructed from such a pattern by
a neural network that is trained on a large number of scattering
events on known objects exposed to a number of different illum-
ation conditions. From the prospective of Fourier optics, topo-
logically structured superoscillatory illumination gives access to
large local wavevectors and ensures that the scattering pattern
of structured light is more sensitive to the small features of the
imaged object than that of the unstructured light. In recent exper-
iments, we show that with topologically structured illumination
an accuracy beyond λ/10 000 (a few tens of femto-meter), can be reached which brings it to the sub-atomic scale.[203]

Although so far, the deeply subwavelength topological mi-
croscopy has been demonstrated for 1D imaging, it can be read-
ily extended to 2D and 3D objects, as well as objects of random
shape by building the image from point-to-point scanning. It will
be most efficient in microscopy of a priori known shapes, such as
those found in routine tasks of biomedical research and machine
vision. Furthermore, the technique does not require labeling of
the sample with luminescent materials, nor intense laser illumi-
nation, and is resilient to noise.

As it has been realized,[206] in practical terms, the main chal-
lenge in the implementation of deep learning-enabled optical
microscopy is creating a reliable and trustworthy training set
for neural network training. Such a dataset can be either virtual
or physical. The virtual training dataset of imaged objects and
their diffraction scattering patterns can be generated by numer-
ical modelling (Maxwell solving) on a random set of a priori de-
fined large set of random or application-specific virtual training
objects. Here, the main challenge is to ensure that the computer
model is congruent with the physical realization of the micro-
scope, which may be problematic. Alternatively, a physical dataset
can be created by fabricating a number of real scattering elements
followed by recording of their scattering patterns in the imaging
instrument. Generating a physical set is labor-intensive, but such
a set is naturally congruent with the imaging microscope. The
choice of the training dataset (physical or virtual) shall be made
with the desired resolution and complexity of the microscope op-
tical tract, and, in general, higher resolution would require large
training datasets. Neural networks can also be trained on hybrid
data sets that are partially physical and partially virtual. An im-
portant direction for future research is the analysis of resolution
limits of microscopies using prior knowledge of the sample and
deep learning techniques that give access to the large amount of
information accrued during the neural network training process
with hybrid sets.

In conclusion, we are at the verge of a new era of microscopy,
the era of Deeply Subwavelength Optical Microscopy. The diffrac-
tion limit and imaging lens will be the things of the past. The
light scattered by the object will be the main source of informa-
tion about it. Illumination with topologically structured light will
make these diffraction patterns highly sensitive to the nanoscopic
features of the sample. The microscopy will not require label-
ing of the samples and engaging the nonlinear regimes of light-
matter interaction. The molecular and even atomic resolution
will be routinely achievable. Imaging of soft-matter biological and
solid-state nanotechnology samples will be equally possible in
one, two, and three dimensions. Imaging at multiple laser fre-
quencies will give access to microscopy in color and the wave-
length range can be readily extended to any part of electromag-
netic spectrum where coherent sources of light and detectors ex-
ist. Microscopy will be based on reconstructing shapes of the scat-
tering object with artificial intelligence and will be limited in res-
olution only by size and quality of the training sets.

High throughput diffraction simulations and Maxwell solving
will be used for creating the training data sets. Training sets with
increased sophistication will be developed. At some level of fi-
nesse we will need to go beyond the effective medium approxi-
mation when material properties are described by their perme-

ability and permittivity, and ab initio modelling of light scatter-
ing on molecular structures will have to be developed to build
the training sets.

It is likely that training of neural network will be application-
specific for tasks such as, for example, protein folding or semi-
conductor chip fault diagnostics. The size of training sets can eas-
ily reach petabyte volumes and may require super-computers to
generate. However, the end-user will not need to deal with Big
Data, they will only need to be informed with the final results of
training, which can be distributed as a file to set the neural net-
work on the imaging instrument computer, a simple desktop PC.
While the increase of resolution will require increased computer
resources, the optical imaging act will remain simple: a single or
multiple shot registration of the diffraction patterns.

5. Label-Free Plasmonic/Metamaterial Structured Illumination Microscopy

(Junxiang Zhao, Zachary Burns, Zhaowei Liu*)

Structured illumination microscopy (SIM) doubles the resolu-
tion of a wide field fluorescent microscope and has seen many
applications in biomedical studies. However, SIM provides no
additional resolution enhancement over oblique illumination for
label-free imaging due to the coherent nature of the scattering
process. Meta-substrates using plasmonic structures and meta-
materials provide a possibility to achieve illumination wave vec-
tors beyond the limit of traditional optics, leading to a unique
path for label-free super-resolution imaging. Here, we review the
current super-resolution imaging techniques that combine SIM
with meta-substrate assisted illuminations. We outline the chal-
lenges in achieving label-free super-resolution imaging and offer
a set of directions for future work.

Optical microscopy has been an irreplaceable tool in biomedie-
cal research but has a limited resolution due to the wave nature of
the light. According to the Abbe diffraction limit of λ/2NA, where
λ is the light wavelength and NA is the numerical aperture of the
optics, the maximum spatial resolution of a conventional lens-
based microscope at visible frequencies is around 200nm.

Numerous super-resolution microscopy (SRM) techniques[14, 20, 22] has been developed for fluorescent microscopies to break the diffraction limit. Among these SRM techniques, structured illumination microscopy (SIM)[26] utilizes a series of periodic illumination patterns superimposed onto an object to improve the resolution by 2-fold. Compared to other SRM techniques, SIM's advantages are its high imaging speed, wide field-of-view (FOV) and low photodamage, representing a nice combination for many biomedical applications.[28] However, since the scattering process is coherent, label-free imaging using scattered light gains little from structured illumination.[93] The bandwidths for detection and excitation are both limited within the coherent transfer function (CTF) using conventional optics. Therefore, for coherent label-free imaging, the structured illumination scheme does not yield any more information than imaging with oblique illuminations, which has a resolution well defined by the Abbe diffraction limit.

SIM achieves super-resolution by frequency mixing the high
spatial frequency components of the object into the detectable
bandwidth of a microscope. The highest achievable resolution
<figure><img src="image_10.png" /><figcaption>**Figure 7.** Concept of resolution enhancement with PSIM and LPSIM. a) The dispersion relation comparison of a propagating photon in dielectric media, a SPP at dielectric/metal interfaces, and the LSP field from a nanoantenna array. The wavevectors at the illumination frequency are denoted $k_p$, $k_{SPP}$, and $k_{LSP}$ respectively. Inset: SEM images of typical PSIM and LPSIM substrates. A PSIM substrate is constructed from thin Ag film with an array of patterned slits. A LPSIM substrate consists of a hexagonal silver disc array. Reproduced with permission.[104] Copyright 2014, American Chemical Society. b,c) Resolution improvement of PSIM and LPSIM for incoherent and coherent imaging system, respectively. The gray circle represents the passband of a conventional microscope. The green circles and yellow circles correspond to the spatial frequency shift due to SPP and LSP illuminations. The dashed circles represent the total detectable Fourier components of the objects with PSIM and LPSIM.</figcaption></figure>

is determined by the sum of illumination and detection spatial frequencies: $f = f_{det} + f_{illum}$. For a coherent imaging system with both illumination and detection cutoff frequency at NA/λ, the maximum attainable resolution is λ/2NA. Techniques like Fourier ptychography[96] use high angle oblique illumination combined with low NA objectives to achieve multiple fold resolution enhancement compared to the low NA resolution limit. But since the NA of oblique illuminations in free space cannot surpass 1, ultimately such methods cannot achieve higher resolution than the Abbe diffraction limit of high NA optics.

Since optical microscopes typically require far-field detection, the detection bandwidth cannot be further improved other than using objectives with the highest available NA. Therefore, creating higher spatial frequency illumination patterns is the most viable and crucial approach to achieving label-free super-resolution SIM. Plasmonic structures and hyperbolic metamaterials are known for supporting larger wavevector waves compared to dielectric medium and have seen many applications in sub-wavelength imaging and sensing.[212-214] By introducing a meta-substrate designed with a plasmonic nanostructure or metamaterial to generate illuminations, metamaterial assisted illumination nanoscopy (MAIN) can achieve much higher resolution than standard SIM methods.[215]

One way to create illuminations beyond the diffraction-limit is to use a meta-substrate containing plasmonic nanostructures. Plasmonic structured illumination microscopy (PSIM)[104] and localized plasmonic structured illumination microscopy (LPSIM)[105, 106] use interference patterns from counter propagating surface plasmon polaritons (SPP) and localized surface plasmon (LSP) waves from metallic nano-disc arrays to generate illuminations beyond the traditional diffraction-limit respectively. **Figure 7a** illustrates the lateral wavevector comparison for propagating light, PSIM illumination and LPSIM illumination. Contrary to traditional SIM, the maximum achievable bandwidths for PSIM and LPSIM with fluorescent imaging are described by $k_{PSIM} = 2k_{SPP} + 2NA \times k_0$ and $k_{LPSIM} = k_{LSP} + 2NA \times k_0$, where $k_{SPP}$ and $k_{LSP}$ are the wavevectors of PSIM and LPSIM illumination

patterns, and $k_0$ is the free space wavevector of the illumination wavelength (Figure 7b). Compared to the diffraction-limit, PSIM has experimentally demonstrated ≈2.6-fold resolution improvement[104] and LPSIM has demonstrated 3-fold resolution improvement for biological samples.[105, 106] For a coherent label-free imaging system with CTF cutoff frequency at NA/λ, the illuminations from PSIM and LPSIM are purely dependent on the nanostructures and thus can achieve higher resolution beyond the diffraction limit described by: $k_{PSIM/LPSIM} = k_{SPP/LSP} + NA \times k_0$ (Figure 7c). At this moment, however, the experimental demonstration of label-free PSIM and LPSIM has yet to be done.

Hyperbolic metamaterials (HMM) have recently been used as meta-substrates for ultra-high resolution optical imaging.[108, 216, 217] **Figure 8a-b** show that an ideal HMM can support arbitrarily large lateral wavevectors as the isofrequency surface of an HMM is a two-sheet hyperboloid for a Type I HMM ($\epsilon_{\perp} > 0, \epsilon_{\parallel} < 0$) or a one-sheet hyperboloid for a Type II HMM ($\epsilon_{\perp} < 0, \epsilon_{\parallel} > 0$). HMMs are commonly implemented by two types of structures: a lattice of metallic nanorods embedded in a dielectric medium or a stack of deep subwavelength alternating-metallic and dielectric layers. For imaging applications, the multilayer structures are the most practical methods to implement the HMMs. A practical HMM consists of multiple layers of metal and dielectric films can support extremely large lateral wavevectors[109] in the near field, thus they can create high resolution illumination patterns (Figure 8c). One implementation of HMM assisted illumination microscopy is hyper-structured illumination,[108, 217] which utilizes the highly dispersive nature of an HMM to project a series of deep-subwavelength illumination patterns which are tuned by the incident wavelength. A 6-fold resolution improvement over the diffraction limit has been reported for hyper-structured illumination with label-free imaging, resolving features down to ≈80 nm.[108] Another implementation of MAIN, named speckle-MAIN,[109] creates deep sub-wavelength speckle illumination patterns by taking advantage of the inherent material roughness within the multilayer HMM (Figure 8d) and achieves 40 nm resolution with a fluorescent microscope.
<figure><img src="image_11.png" /><figcaption>**Figure 8.** HMM for super-resolution imaging. a) The isofrequency surface for a Type I HMM ($\epsilon_{\perp} > 0, \epsilon_{\parallel} < 0$). Type I HMMs are commonly achieved with nanorod arrays. b) The isofrequency surface for a Type II HMM ($\epsilon_{\perp} < 0, \epsilon_{\parallel} > 0$). Type II HMMs are commonly fabricated with metal-dielectric multilayers. c) Transmission of a multilayer HMM with 12 pairs of 5 nm Ag and 5 nm SiO$_2$. d) Calculated exemplary speckle patterns for a diffraction limited system (left) and HMM assisted speckles (right). Scale bar, 1 µm.</figcaption></figure>

Recently, novel organic hyperbolic metamaterials (OHM) fabricated by self-assembled regioregular poly(3-hexylthiophene-2,5-diyl) have been demonstrated to support remarkably large lateral wavevector in the visible frequencies.[218, 219] The OHM assisted MAIN produces ultra-high resolution of 30 nm while exhibiting superior biocompatibility and photostability due to its exceptionally large Purcell factor at the film surface.[219, 220]

One of the main challenges of label-free super-resolution imaging is the low photon budget due to the very weak scattering of sub-wavelength objects. The resulting low signal-to-noise ratio (SNR) causes significant loss of high-resolution information as higher spatial frequency components of an object typically have very low spatial spectrum density. Recently, plasmonic scattering imaging[221] demonstrated imaging of single proteins by using a thin gold film as the sample substrate. With a meta-substrate, significant enhancement of scattering from small objects greatly improves the SNR and allows for far-field detection. This is especially crucial for imaging tiny objects with a target resolution of sub-50 nm, as the scattering cross-section is too small to create a meaningful measurement with traditional imaging methods.

Image reconstruction has also been a major roadblock for realizing label-free MAIN. Fluorescent versions of MAIN that rely on intensity fluctuation based reconstruction methods[222, 223, 24] have shown promising imaging results. However, since scattering is a coherent process, the label-free imaging system is linear with respect to the electric field rather than the intensity. As we detect only the intensity with a camera, an ill-posed non-linear inverse problem must be solved to retrieve the object information. One method to overcome this problem is to use a meta-substrate made of turbid medium and record the transmission

matrix of all near-field input modes at the cost of requiring near-field scanning of the entire FOV prior to imaging. The far-field scattered light of the object will be a linear combination of the basis transmission matrix such that a super-resolution image can be retrieved by time reversal reconstruction.[224]

Increasingly deep neural networks are being used for super-resolution microscopy reconstruction[187, 225] to boost performance over traditional methods, for example by reducing the number of sub-frames and handling cases of low signal-to-noise ratio. Often there is knowledge of the physical system that could be incorporated into the network to reduce the training burden and improve results.[56] Recently there has been the development of “physics-based” or “untrained” neural networks that incorporate a physical forward model into the loss function for training. These methods have been applied to cases such as phase imaging and lensless imaging.[226–228] For label-free MAIN, where the illumination field and forward process is known, there is the potential to incorporate this knowledge into a physics-based network trained on a collection of data. Such a model could combine the benefits of a traditional physics-based model with those of statistical learning to create better reconstructions and alleviate some of the challenges of the ill-posed inverse problem.

Current label-free MAIN results are, at this point, mostly preliminary. Hyper-structured illumination has achieved label-free super-resolution for a 1D object, but its 2D imaging capability has not been demonstrated. PSIM, LPSIM and speckle-MAIN have only been experimentally demonstrated with fluorescent imaging and a robust image reconstruction scheme for coherent imaging has yet to be developed. Nevertheless, MAIN addresses the major difficulties in achieving super-resolution label-free
imaging with its exceptionally larger illumination bandwidth and SNR improvement over traditional optics.

# 6. Fundamental Limits to the Resolution of Label-free Far-field Microscopy

(Evgenii Narimanov*)

## 6.1. Introduction

High resolution optical imaging is essential for the investigation of fundamental microscopic processes both in nature and in artificial systems – from the charge carrier dynamics in electronic nano-circuits[229] to the biological activity in cellular structures.[230] However, as optical diffraction prevents focusing the light into a spot with dimensions much smaller than its wavelength,[231] the resulting Abbe’s diffraction limit[232–234] does not allow the conventional optical microscopy to directly image subwavelength structures such as cell membranes, individual viruses, or large protein molecules. As a result, there is now an increasing interest in “super-resolution” optical methods that allow to overcome this diffraction barrier – with a particular demand for a label-free approach that is inherently non-toxic and operates on the sample that is in the far-field from all elements of the imaging system.

Recent developments in label-free optical super-resolution imaging[132, 235–237, 52, 53] clearly demonstrate that Abbe’s bound of half-wavelength (and its quarter-wavelength counterpart for structured illumination) is not a fundamental limit for optical imaging. Nevertheless, there is in fact a fundamental limit on the resolution of far-field optical imaging, but it is much less stringent than Abbe’s criterion. While is a perfectly noise-free environment arbitrary resolution is possible,[238] the presence of any finite amount of noise in the system leads to a limit on the optical resolution, that has its origin in Shannon’s theory of information transmission in linear systems.[239]

## 6.2. The Definition of the Resolution Limit

We define the resolution limit $\Delta$ as the shortest spatial scale of the object whose geometry can still be reconstructed, error-free, from the far-field optical measurements in the presence of noise.[60] The essential “lower bound” nature of $\Delta$ then allows to reduce the problem to the limiting case of an object that is composed of an arbitrary number of point scatterers of a priori unknown amplitudes located at the nodes of the grid with the period $\Delta$, as any additional structure in the sources (or scatterers) as well as any further variations in their positions, will only add to the information that needs to be recovered from the far-field image. Furthermore, the same argument allows to reduce the task of calculating $\Delta$ to the case of an effectively one-dimensional target (formed by line, rather than point, sources) – since, as it was already known to M. André[240] and L. Raleigh,[241] line sources are “more easily resolvable” than point sources.

Note that, even within the original framework of “resolving” two point sources,[241] the resolution clearly depends on the difference of their amplitudes – with increasing disparity between

the two leading to progressively worse “performance”. The “ultimate” resolution limit $\Delta$ therefore corresponds to the case of identical point sources/ scatterers, which are present only in an (unknown) fraction of the grid nodes.

For the definition of the fundamental resolution imaging, it is essential to assume coherent detection, as the phase information is generally available in the far field, and can always be measured even with an in- tensity only sensitive detector using e.g., optical heterodyne approach,[242] – so that any failure to obtain the phase information in a given experimental setup cannot be attributed to a fundamental limitation.

## 6.3. Information-Theoretical Framework

The fundamental limit on the resolution of optical imaging is reached when the total amount of information about the object that has been recovered in the far field, is equal to the amount of information that is necessary and sufficient to define the geometry of the object. Treating the optical point-spread function of the imaging system in the noisy environment as the input – output relation of a linear information channel, the amount of the actual information carried from the object to the far field detector, can be obtained using the standard methods of the information theory,[239] in terms of the mutual information functional, calculated between the statistical distributions of the object profile and of the far field scattering pattern, as described in detail in Ref. [60] The resolution limit then follows from the requirement of the recovered information being sufficient to reconstruct the target:

$$ \Delta = \frac{L}{T} \qquad (5) $$

where $L$ is the length of the object, and $T$ is the information about the structure and geometry of the object that was successfully transmitted to the detector(s) at the far-field. Note that when the object is composed of $M$ different materials (or is formed by an array of point sources with $M$ different levels of amplitude), additional information is needed for its reconstruction, which leads to a more stringent bound on the spatial resolution,

$$ \Delta_M = \frac{L \log_2 M}{T} = \Delta \cdot \log_2 M \ge \Delta \qquad (6) $$

The transmitted information $T$ can be obtained from the standard mutual information functional[239]

$$ T = H - H_c \qquad (7) $$

where $H$ is the information entropy for the signal at the detector array. As the imaging system is noisy, for any output signal, there is some uncertainty of what was the originating optical field scattered (or emitted) by the object. The conditional entropy $H_c$ of the signal at the detector array for a fixed optical field from the object, represents this uncertainty in Equation (7). Calculating the resulting analytical expressions[239] for $H$ and $H_c$ and substituting them into the mutual information functional $T$ in Equation (7),
<figure><img src="image_12.png" /><figcaption>**Figure 9.** Super-resolution object reconstruction for a binary mask (with the profile shown in the inset), with the error probability of the recovered geometry $P_{err}$ shown as a function of the effective signal-to-noise ratio, SNR$_{eff}$. The boundary separating the gray and light-yellow background, represents the value of the signal-to-noise ratio that according to Equation (8) is sufficient to resolve the $\lambda/16$ spacing (see the inset). The red arrow indicates the minimum value of SNR$_{eff}$ where the numerical reconstruction procedure implemented in[60] shows no errors.</figcaption></figure>

for the resolution limit in the case of uniform illumination, we obtain[60]

$$ \Delta = \frac{\lambda}{2} \frac{1}{\log_2 \sqrt{1 + 2 \text{SNR}_{\text{eff}}}} \quad (8) $$

Here SNR$_{eff}$ is the effective signal-to-noise ratio[60] measured at the detector array, and $\lambda$ is the wave-length. Similarly, in the case of conventional structured illumination[26] (where the illumination pattern is formed in the far-field of the light source(s) and is therefore subject to the diffraction constraints), for the resolution limit we find

$$ \Delta^{\text{SE}} = \frac{\lambda}{4} \frac{1}{\log_2 \sqrt{1 + 2 \text{SNR}_{\text{eff}}}} \quad (9) $$

### 6.3.1. The Discussion

While Equation (8) allows for an unlimited resolution in a noise-free environment, even a relatively low noise dramatically alters this picture. With the weak logarithmic dependence of the resolution limit on the SNR in Equations (8), (9), to reduce the resolution limit by e.g., a factor of ten, the signal-to-noise ratio needs to be increased by six orders of magnitude.

The experimentally observed resolution can reach the value of $\Delta$ in the case of imaging a binary mask or a pattern of identical subwavelength particles – leading to practical imaging well below the Abbe’s limit. However, this can only be achieved in the regime of extreme noise suppression, with the corresponding values of the SNR on the order of $\approx 10^5$ and beyond.

This is illustrated in **Figure 9**, where the profile of a subwavelength binary mask (see the inset to Figure 9) is recovered from its image in the far-field. While a finite amount of noise in the far-field measurements inevitably leads to errors, with the increase of the effective signal-to-noise ratio SNR the corresponding error

probability $P_{err}$ rapidly goes to zero. In particular, for the resolution of $\lambda/16$ in the example of Figure 9, the numerical calculation with an ensemble of multiple ($\approx 10^4$) different realizations,[60] showed no errors for SNR beyond the value indicated by the red arrow ($\approx 10^6$).

However, for many practical imaging applications, especially in the field of biomedical imaging, the level of noise reduction that is needed for subwavelength resolution, is beyond the reach of current technology. For the signal-to-noise level of $\approx 10^2$–$10^2$ typical for a biological environment, we find the ultimate resolution of the order of $\approx \lambda/10$. As a result, any endeavor to push the observed resolution beyond this level in a conventional (direct rather than statistical, and single exposure) imaging, is akin to an attempt to develop a working example of a perpetual motion machine.

This situation is however dramatically reversed when additional information about the object is available a priori. This a priori knowledge reduces the amount of information about the object that must be obtained from its far-field image – which leads to the corresponding reduction of $\Delta$ in the resolution limit of optical imaging. Furthermore, this effect is described by an algebraic rather than logarithmic function. For this reason, rather than improving the noise performance of the imaging set-up, the route to a practical implementation of far-field super-resolution lies in taking full advantage of the a priori information about the target.

Among different examples of a priori available information about the object, particularly important is the case of sparse objects, as this property is widely spread in both natural and artificial systems.[230] When the target is sparse, with the effective sparsity parameter $\beta$ (which can be defined as the fraction of empty “slots” in the grid superimposed on the target), we find[60]

$$ \Delta^{(\beta)} = \Delta \cdot |\beta \log_2 \beta + (1 - \beta) \log_2 (1 - \beta)| \quad (10) $$

For the numerical example studied in Ref. [52], with $\beta \approx 0.03$ and SNR $\approx 10^2$, the resolution limit $\Delta^{(\beta)} \approx 0.025 \lambda$. Accurate numerical reconstruction of the features on the scale of $\approx \lambda/10$ demonstrated in Ref. [52], is therefore fully consistent with the fundamental resolution limit presented here.

Finally, we note that subwavelength resolution of statistical localization microscopy[22, 243, 244] also originates from this mechanism, as each of the multiple images that are assembled in this approach to reconstruct the topology of the target, are measured with extremely low values of the sparsity $\beta$ (when only one or a few of the point light emitters introduced into the object, are currently active), thus according to Equation (10) leading to $\Delta \approx \lambda\beta \ll \lambda$.

### 6.4. Conclusions

In conclusion, we have presented the fundamental resolution limit for far-field optical imaging and demonstrated that it is generally well below the standard half-the-wavelength estimate. Our result also applies to other methods that rely on wave propagation and scattering – such as e.g., geophysical and ultrasound imaging.
# Theme II: Interferometric and Holographic Approaches

## 7. Current Trends In Quantitative Phase Imaging

(Neha Goswami*, Gabriel Popescu†)

Dedicated to the memory of Gabriel Popescu (deceased June 2022)

### 7.1. Introduction

With its 400-year-old history, optical microscopy has become the most widely used technology in biomedicine.[245] Fluorescence-based microscopy is currently dominant, as it provides microstructure information with chemical specificity. However, this capability often comes at a heavy price: chemical and phototoxicity affects the function and viability of live cellular systems and photobleaching drastically narrows down the temporal window of investigation. For these reasons, we have witnessed a renewed interest in label-free imaging, as evidenced by the various sections of this current Roadmap article. In this section, we describe quantitative phase imaging (QPI), an approach to label-free imaging that is rooted in holography and interferometric microscopy.[246]

In recent years, QPI has registered significant progress both in technology development and biomedical applications.[247] It is becoming an important tool in biomedicine due to its label-free and quantitative nature and has proved to be a valuable tool in many areas, including cell biology, blood testing, neuroscience, clinical diagnosis, fertility, and infectious diseases.[247] As a non-destructive technique, QPI enables long-term investigation of live systems. Due to the linear relationship between the phase shift map of a biospecimen and its dry mass density, QPI can report on cell growth and intercellular mass transport with very high sensitivity.[247] Importantly, knowledge of the complex image field is equivalent to that of the scattered field, which allows solving scattering inverse problems and tomographic reconstructions, based on QPI data.[248]

Despite tremendous progress in recent years, which resulted in robust instruments capable of user-friendly operation, we identified several current challenges that can turn into enormous opportunities in the field. These challenges are: achieving super-resolution QPI, performing QPI in thick, strongly scattering tissues, and performing QPI with chemical specificity. In addition to hardware developments to tackle these problems, an exciting, newly emerging solution is artificial intelligence. In this chapter, we review the QPI principles of operation, its figures of merit, applications, and these current challenges and opportunities.

In the 1930's, phase contrast (PC) microscopy was invented as an "intrinsic" method that allowed the visualization of transparent structures with high contrast.[249] The intensity distribution of an image can be described as an interferogram, following Abbe's insight from 1873.[1] Thus, the intensity of a PC image can be written as

$$I(x, y) = |a| I_0 + I_1(x, y) + 2 \sqrt{(|a| I_0 I_1(x, y))} \cos [\Delta\phi(x, y) + \gamma] \quad (11)$$

In Equation (11), $I_0$ and $I_1$ are the intensities of the incident and scattered light, respectively, $\Delta\phi(x, y)$ is the phase difference between the two fields, while $a = |a|e^{i\gamma}$ is a complex filter applied by Zernike to the incident light. The point of the filter is to make $I$ more sensitive to $\Delta\phi$, i.e., to increase the contrast of the interferogram. First, the filter attenuates $I_0$ ($|a| < 1$), which otherwise is much stronger than $I_1$ for transparent structures, resulting in low contrast interference. Second, the phase shift $\gamma$ is set to a value of $\pm\pi/2$, which converts the slowly varying cosine into a rapidly varying sine, which, for small values of $\Delta\phi$ approximates a linear function. With this simple filter, the contrast of the image is increased significantly, which allowed PC to become a broadly adopted technique, especially for visualizing unlabeled cell cultures. However, as can be seen from Equation (11), a single measurement of $I(x, y)$ is insufficient for decoupling $I_0$, $I_1$ and $\Delta\phi$, which means that it cannot be used for quantitative measurements.

A decade later, Gabor invented holography to store phase information into a single intensity recording on a transparency, which can be further “read” with the same incident light and recreate the phase and amplitude information associated with the object field.[250] The equation for describing Gabor’s “in-line” hologram is very similar to Equation (11), except $I_1(x, y)$ corresponds to the scattered light at an out of-focus-plane, described by Fresnel’s formula, and there is no filter, i.e., $a = 1$. Therefore, in-line holography cannot be used to extract quantitative phase information either. Furthermore, because the recorded signal $I$ is real, the reconstructed signal, i.e., the complex field associated with the object is Hermitian, resulting in a pair of images along the optical axis, complex-conjugated of one another. This obstacle become known as the “twin image” problem and was solved later with the advancement of off-axis holography.[251] The goal in QPI is to eliminate all these ambiguities and measure quantitatively the phase shift at each point in the field of view. The main approaches to accomplish this are presented below.

### 7.2. Principles of QPI

When choosing a certain method for QPI, one must be mindful of the figures of merit that are most important for the application at hand. There are four main parameters that describe the performance of a QPI system: spatial and temporal resolution, as well as spatial and temporal phase sensitivity.[246] Phase shifting interferometry (Figure 10a) is one solution for decoupling the phase and amplitude contributions in Equation (11). By controlling the phase difference between the two interfering fields, one records several independent interferograms, which solves for the phase uniquely. As a time-modulation technique, phase shifting preserves the space-bandwidth product of image, at the expense of the time resolution.

Conversely, off-axis interferometry (Figure 10b) isolates the cross-term in Equation (11) by spatial modulation and, as a result, provides high acquisition rates at the expense of space-bandwidth product.

The spatial and temporal sensitivity describe the smallest phase changes that an instrument can detect between two points, respectively, in space and time. The temporal sensitivity is affected by the stability of the instrument and techniques for
<figure><img src="image_13.png" /><figcaption>Figure 10. QPI modes of operation. a. Phase shifting interferometry b. Off-axis interferometry. $k_i$ is the incident wavevector and $k_r$ is the reference wavevector. $U_1(x, y)$ is the scattered component of the complex field $U(x, y)$ at the image plane.</figcaption></figure>

noise cancellation. Common-path interferometers are intrinsically most stable. (see Chapter 8 in Ref. [246]) Spatial sensitivity is dictated by the nonuniformity within a field of view in the absence of a sample. Suppressing speckles by using both spatial and temporal frequency averaging via, respectively, a rotating diffuser or broadband illumination is critical in achieving high spatial sensitivity.[247]

### 7.3. Status: Applications of QPI

It has been discovered early on that a quantitative phase map $\phi$ is linearly related to the surface dry mass density of the structure, $\rho$,

$$\rho(x, y) = \frac{\lambda}{2\pi\alpha}\phi(x, y) \qquad (12)$$

The proportionality constant, $\alpha$, referred to as the refractive increment, is relatively constant across a variety of molecular species within live cells, $\alpha \approx 0.2$ mL/g.[246] This simple relationship allows researchers to “weigh” cells by simply imaging them and study cell growth and its modulation nondestructively, over many cell cycles.[252]

Time-lapse QPI data yields dynamic dry mass density maps, $\rho(x, y, t)$, which in turn can be used to understand the physics of the intracellular transport. This transport has diffusive (passive, random) and directed (active, deterministic) contributions, and is governed by the diffusion-advection equation,[253]

$$(Dq^2 - i\mathbf{q} \cdot \mathbf{v} + \frac{\partial}{\partial t}) \rho(x, y, t) = 0 \qquad (13)$$

where $D$ is the diffusion coefficient, $v$ is the velocity, and $q$ is the spatial frequency.

Assuming a distribution of velocities of width $\Delta v$, the dispersion relation associated with Equation (13) yields

$$\Delta\omega(q) = Dq^2 + \Delta vq \qquad (14)$$

Equation (14) indicates that measuring the temporal bandwidth as a function of spatial frequency, $\Delta\omega(q)$, reveals the diffusion component as a quadratic dependence and the active one as a linear relation. It has been found that intracellular transport for most cells is characterized by a diffusive component at short distances (large $q$) and a directed one over large distance (small $q$ values) (Figure 11a).[253] This result indicates that cells optimize their energy consumption and involve the molecular motors to perform work only for transport at larger scales of several microns.

Highly sensitive QPI data has been shown to hold tremendous potential in cancer pathology.[257] Thus, phase maps of histology slides reveal not only the epithelial structure of tissues, which can be used for label-free diagnosis (Figure 11b),[247, 258] but also the tumor microenvironment, which can provide information about the disease aggressiveness, (Figure 11c).[254] QPI has been employed to quantify the alignment of collagen fibers, a task that previously was possible only by using specialized stains or second-harmonic generation microscopy. Collagen arrangement has been shown to be a valuable marker in predicting metastasis and patient survival outcome.[254]

Using QPI data, one can reconstruct the tomography of the refractive index distribution, as shown by Wolf's classical paper in 1969.[248] Under the weakly scattering approximation, the scattering potential, $F$, is just the 3D Fourier transform of the scattered field, $U$, $F(q) \propto U(q)$, where $q = (2\pi/\lambda)\sin(\theta/2)$ is the scattering wavevector and $\theta$ is the scattering angle. Thus, the 3D reconstruction requires collecting QPI frames along a third dimension, whether direct z-axis, or equivalently, illumination angle or wavelength. The understanding that QPI data is entirely equivalent to those obtained via angular scattering, led to the development of tomographic methods that are robust and ready for wide adoption in the biomedical field[Figures 11d and e and Refs.[255, 256, 259–262]]

### 7.4. Challenges and Advances in QPI

There is no doubt that the tremendous effort in the QPI field during recent years resulted in technology solutions to seemingly insurmountable problems. Achieving sub-nanometer pathlength stability over several days appeared out of reach in the early 2000's. Now, many variants of such instruments exist and are also being commercialized globally. Developing an instrument so easy to use and robust that it would allow broad-scale use by non-engineers appeared far-fetched as well. Yet, today, we see commercially available QPI systems that are increasingly embraced by biologists and clinicians world-wide. The application range of this technology has been growing exponentially, branching out from cell biology into infectious diseases,[263] neuroscience,[264] tissue engineering,[265] fertility,[266] pathology[267] and beyond.[268] We anticipate that applications to non-biological field, such as materials and polymer science will also continue to grow.[269]
<figure><img src="image_14.png" /><figcaption>**Figure 11.** Applications of QPI. a) QPI reveals intracellular cell mass transport: Quantitative phase image of a culture of glia (left image) and Dispersion curves (right image), Γ(q), in log-log scale, associated with the white box regions in left image, respectively. The green and red lines indicate directed motion and diffusion, respectively, with the results of the fit as indicated in the legend. Reproduced with permission.[253] Copyright 2011, Optica Publishing Group. b) Normal colon epithelial nuclei from an ulcerative colitis patient who did not develop colorectal cancer after seven years (low risk, top panel) and from an ulcerative colitis patient who developed colorectal cancer after seven years (high risk, bottom panel). Scale bars 20 µm. Color bars indicate optical path length in nm. Reproduced under terms of the CC-BY license.[247] Copyright 2018, The Authors, published by Springer Nature. c) Collagen fiber orientation probability densities and bar charts with counts of isotropic and anisotropic regions for three different cores. Scale bar: 200 µm. Reproduced with permission.[254] Copyright 2017, SPIE d) 3D rendered isosurfaces of RI maps of individual healthy RBC. Reproduced with permission.[255] Copyright 2014, The Authors, published by Springer Nature. e) Top: a deconvolved z-slice measured using a ×63/1.4 NA oil immersion objective. Middle: a cross-section at the area indicated by the red box and the yellow box. Bottom: false-color three-dimensional rendering of the deconvolution result. Scale bars in all panels, 5 µm. Reproduced under terms of the CC-BY license.[256] Copyright 2014, The Authors, published by Springer Nature.</figcaption></figure>

However, to continue this tremendous progress and eliminate further obstacles that may prevent QPI from becoming a routine method of investigation in biomedicine, we must assess critically the current limitations of the technology. In our opinion there are three major shortcomings of the QPI technology: 1) imaging strongly scattering tissues is extremely challenging, 2) as a label-free technique, QPI lacks chemical specificity, and 3) as a diffraction limited method, QPI has limited resolution. Solving these problems represent enormous opportunities in the field, as briefly described below.

### 7.4.1. QPI of Strongly Scattering Structures

It has become increasingly clear that 3D cellular clusters better mimic the behavior of in-vivo tissue. Such systems are crucial not only in tissue engineering, but also for developing "biobots" and precision medicine.[270] As a label-free, nondestructive tool, QPI appears ideal for investigating such cellular systems in terms of function and viability. However, as these structures can range from 100s of microns to millimeters in size, they are significantly larger than the scattering mean-free path and thus, produce
<figure><img src="image_15.png" /><figcaption>**Figure 12.** Current trends in QPI. a) Phase imaging with computational specificity (PICS)-To demonstrate time-lapse imaging and high-content screening capabilities, the authors of Ref. [272] seeded a multiwell with three distinct concentrations of SW cells (×20/0.8). These conditions were imaged over the course of a week by acquiring mosaic tiles consisting of a 2.5 mm² square area in each well using a ×20/0.8 objective. The machine learning classifier, trained at the final time point after paraformaldehyde fixation, is applied to the previously unseen sequence to yield a Dil and DAPI equivalent image. Interestingly, the neural network was able to correctly reproduce the Dil stain on more elongated fibroblast-like cells, even though few such cells are present when the training data was acquired (white arrows). Reproduced under terms of the CC-BY license.[272] Copyright 2020, The Authors, published by Springer Nature. b) Phase-stain for image-to-image translation-PhaseStain-based virtual staining of label-free kidney tissue (Jones' stain) and liver tissue (Masson's Trichrome). Reproduced under terms of the CC-BY license.[57] Copyright 2019, The Authors, published by Springer Nature. c) Resolution enhancement through deep learning-Spatial frequency analysis for the diffraction-limited system. Reproduced under terms of the CC-BY license.[273] Copyright 2019, The Authors, published by Springer Nature.</figcaption></figure>

strong multiple scattering. The multiply scattered waves, characterized by a distribution of propagation direction and phase shifts, overlap at each point in the field of view creating in essence a strong incoherent background. As a result, the singly scattered light, carrying the phase image information, is subdominant, overwhelmed by the multiple scattering background. That is to say, the image recorded exhibits low contrast. Initial efforts to tackle this challenge already have been made: using a DIC geometry, in which the interfering fields are always equal in power and phase shifting that only modulates the single scattering light, GLIM has shown promise for imaging embryos and spheroids.[271] However, as a wide-filed QPI method, GLIM still suffers from “spatial cross-talk”, whereby scattered light from neighboring points overlaps in the same field of view.

This tremendous challenge is likely to generate new efforts in the field for the foreseeable future. While suppressing multiple scattering to increase penetration depth is a problem applicable to all optical techniques, label-free or otherwise, the separate access to amplitude and phase specific to QPI, may provide the path toward viable solutions. Along with new technology development, we anticipate that improved physical models coupled with computational algorithms will contribute towards suppressing or us-

ing multiple scattering for 3D tomography of thick objects. In particular, deep learning models may augment the physics-based descriptions for this purpose.

### 7.4.2. Phase Imaging with Computational Specificity (PICS)

Image to image translation based on deep learning has shown enormous potential for converting computationally one form of contrast into another.[57] Using label-free data as input, it has been shown that specific fluorescence or stain maps can be inferred (Figures 12a-b).[272, 57] While still in its infancy, phase imaging with computational specificity (PICS)[272] has demonstrated its value in identifying subcellular and tissue compartments with high specificity but without the shortcomings associated with the fluorescence dyes. This approach virtually has an unlimited range of potential applications, where toxic dyes are replaced by harmless computational algorithms that can operate in real time, without concerns about photobleaching. Because the data is recorded via QPI instead of fluorescence, which often yields weak signals, the exposure and acquisition rate are significantly better. We envision that the ideal QPI device of the future will have its
seamless integration with fluorescence and enable “learning” of various chemical signatures automatically.

### 7.4.3. Improving Resolution of QPI Systems

Existing QPI systems are diffraction limited, with a limited bandwidth coverage that is typically different in the axial vs. transverse direction. The resolution of a microscope is limited by the numerical aperture (NA) of the imaging optics. For a partially coherent illumination, the transverse resolution can be expressed as[274]

$$r = \frac{1.22\lambda}{(NA_{obj} + NA_{cond})} \Big|_{NA_{cond} \le NA_{obj}} \quad (15)$$

where, NA<sub>obj</sub> is the NA of the objective lens and NA<sub>cond</sub> is the NA of the illumination condenser lens.

Resolution enhancement for QPI has been a subject of current research.[275] Resolution improvement utilizing partially coherent illumination and high NA optics has been reported in literature for 3D refractive index distribution measurement.[271, 276, 277]

Oblique illumination can synthetically increase the NA of the system and hence improve the resolution.[278] The resultant “synthetic aperture” can thus capture higher frequencies associated with the object thereby increasing the resolution.[279, 96] Enhancement in resolution by incrementing NA of the digital holographic microscopy (DHM) system using 2D hexagonal phase grating and recording holograms which are spatially multiplexed has been reported as another potential candidate in superresolution QPI.[280]

An important development is introduced in Ref. [279] where quantitative phase information is retrieved with improved resolution, directly from a set of intensity images by applying space Kramers-Kronig relationship and synthetic aperture approach using oblique illumination. This concept was also shown to be applicable to the optical diffraction tomography for estimating the 3D refractive index distribution.[279] QPI techniques based on Fourier ptychography[96] and intensity based diffraction tomography have also shown an increase in the resolution of 3D refractive index distribution by using an LED array as a source of illumination followed by phase recovery through an algorithm utilizing Born-approximation.[281] Another intensity based diffraction tomography method using annular LED illumination has reported gain in acquisition speed while maintaining near diffraction-limit spatial resolution in 3D.[282]

Resolution enhancement by structured illumination has been another popular method reported in literature.[283] Here, sample is illuminated with a structured light (for example, a sinusoidal pattern) which can then shift the higher frequencies of the object (which were originally outside the NA of the objective) into the existing aperture of the imaging system, thereby increasing the resolution.[284]

Phase deconvolution microscopy based on the principle of deconvolution of through focus stack of images by experimental/theoretical model of point spread function (PSF) has also been shown to increase the resolution and signal to noise ratio (SNR) in 3D QPI.[285, 286]

Highly desirable goal in this area of research is thus to achieve the definition of “superresolution”, which strictly refers

to diffraction-unlimited resolution, as achieved in fluorescence microscopy via nonlinear light-sample interactions.[287]

Recent efforts have shown that, again, deep learning may hold the potential for filling in the missing frequencies and extend the frequency range to super-resolution domain (Figure 12c).[273] The high signal to noise measurement of the complex field in QPI can increase the chances of this frequency extrapolation beyond what is currently achieved with deconvolution methods. This form of image-to-image translation, where the output consists of a super-resolved version of the input, will benefit from a new generation of instruments combining QPI and one of the established fluorescence super resolution techniques, such as STED, PALM, or STORM.

## 7.5. Conclusion

In this section we aimed to present a brief overview of the QPI field, its principles of operation and applications. While acknowledging the incredible progress that QPI has experienced in recent years, we also point out the three main challenges that we believe to hold the key to the next wave of opportunities in the field. It is very likely that, with new advances in computational specificity, thick tissue imaging, and resolution, QPI will be embraced by the biomedical community at large.

# 8. The Present and Future of Scattering Microscopy for Studying Biomolecular Mechanisms

*(Emanuel Pfitzner, Philipp Kukura*)

## 8.1. Status

Proteins are the fundamental building blocks of life, involved in almost all biological processes. The underlying molecular mechanisms invariably rely on interactions between proteins. These interactions are analogous to chemical reactions, where covalent bonds between atoms are formed or broken. They range from simple protein-protein interactions, such as those between antibodies and antigens, to more complex biochemical reactions – like protein biosynthesis, G-protein coupled receptor (GPCR) signalling, and virus-cell interactions (Figure 13). Revealing the structural and dynamic aspects of these processes is key not only to our understanding of biomolecular function and regulation, but also to devise therapeutics for disease intervention[288] and to the creation of novel bio-inspired materials and processes.[289]

Our ability to study biomolecules has been revolutionised by the development of ever more powerful structural methods, which have recently been complemented by AI-based structure prediction.[290] However, our ability to observe and thus study biomolecular mechanisms at the molecular level is much less advanced; insights into mechanisms are generally extracted from a series of experiments involving biochemical and biophysical techniques, combined with structural information. Single molecule methods – predominantly based on single molecule fluorescence, atomic force microscopy or optical trapping – have contributed significantly to progress in this area. Nonetheless dynamics, and the choreography of protein-protein interactions
<figure><img src="image_16.png" /><figcaption>**Figure 13.** Importance of protein-protein interactions and their relationship to polarizability and mass. Counter-clockwise from top left: oligomerisation, antibody-antigen interactions, protein stability, viral infection, protein synthesis, G-protein coupled receptor signalling.</figcaption></figure>

that actually manifest the mechanisms of interest, remain diffi-
cult if not impossible to observe directly.

Label-free detection of single biomolecules by light scattering is attractive for studying protein-protein interactions in a close-to-native environment because of its universality, but has struggled in both development and application due to the associated weak signals. To overcome this limitation, early studies exploited plasmonic amplification of the electromagnetic field,[291, 292] or inelastic scattering.[293] More recently, elastic scattering has been shown to enable single molecule sensitivity using both plasmonic structures,[221, 294, 295] and Rayleigh scattering.[296–298] For interferometric detection of elastically scattered light, the resulting signals are proportional to the polarizability $\alpha_i$ of the scatterer, which has been shown to correlate exceptionally well with its molecular mass $m_i$ (Figure 13).[299] As a result, interferometric scattering data add critical – and universal – information about molecular structure, composition and identity to a pure detection modality.

**8.2. Current Challenges and Required Advances**

The key challenge associated with detecting and quantifying single proteins and their complexes with elastically scattered light arises from their small size (2–20 nm) and consequently minuscule scattering cross sections, and from their refractive indices being similar to water. A typical 50 kDa protein will only scatter a small number of photons in one ms, even at high incident power densities of 1 MW cm⁻². The weak scattering imposes a hard limit on detection sensitivity and the precision achievable when relying on scattered photon detection. Even if the challenge of collecting sufficient scattered photons is met, however, specific detection always has to be achieved on top of any background signatures from spurious reflections, sample inhomogeneities or signals created by other scatterers. The required advances can

be coarsely summarised as a quest for more scattered light, extended detection time, improved statistics and additional information beyond scattering alone.

**8.2.1. More Light**

Particle contrast can be amplified by either boosting the total number of detected photons to reduce shot noise (Figure 14a) or by increasing the number of scattered photons relative to a reference field (Figure 14b). Both can be achieved by higher incident power densities, ultimately limited by the need to prevent sample damage. An alternative approach involves confining the incident light by means of total internal reflection (TIR),[300] wave guides,[298] or dielectric or plasmonic[221] nanostructures. The confinement can boost the local electric field amplitude from two-fold (TIR) to beyond 10-fold for plasmonic structures, although it may be that such increases in local field strength could damage the sample and are subject to ultimately the same limitations as far-field approaches as they increase the local intensity experienced by the analyte. At the same time, resonance enhancement could result in more significant energy deposition – causing damage at lower intensities.

**8.2.2. More Time**

Current approaches to detecting single proteins by light scatter-
ing rely largely on detecting individual proteins after binding to
a surface. Since the change in contrast is too small to be directly
detected, a ratiometric approach is employed: several frames
before and after a landing event are separately averaged, and the
two resulting images are divided. Consequently, every protein is
only measured once. The precision of a contrast measurement of
one single protein could be significantly improved if it could be
<figure><img src="image_17.png" /><figcaption>**Figure 14.** Future challenges of light scattering-based microscopy. a) Detector improvement for lowering shot noise. b) Amplification of light scattering. c) Prolonged, repeated observation. d) Larger field of view for improved statistics. e) Improving measurement precision of individual molecules (#1, #2, #3) by higher statistics per molecule. f) Passivated/activated surfaces for selective measurements. g) Image processing for improving image quality. h) Classification algorithms or additional experimental information (fluorescence, polarisation, etc.) to differentiate otherwise indistinguishable particles.</figcaption></figure>

performed multiple times (Figure 14e), akin to efforts in charge
detection mass spectrometry.[301] Making multiple interfero-
metric scattering measurements requires that the molecule be
mobile and distinguishable from the background multiple times.
One route to achieve this is to adsorb proteins to a solid supported
lipid bilayer on which they can diffuse in 2D (Figure 14c).[302, 303]
or through the use of nanofluidics to achieve 3D trapping.[304]

8.2.3. More Particles

The lack of specificity and temporal resolution for light
scattering-based detection limits the number of particles that can
be detected per unit time. One approach is to devise optical se-
tups with a larger field of view (Figure 14d). Both longer and in-
creased sampling of individual molecules will improve the pre-
cision of the measurement, making it possible to distinguish be-
tween species of similar mass (Figure 14e). Alternatively, selec-
tive surfaces can help by allowing only one species in a complex
mixture to bind while rejecting others (Figure 14f). Enhanced
image processing algorithms, which have been successfully ap-
plied e.g. to transmission electron microscopy (TEM) data, could
help by improving measurement precision through classification
rather than relying on independent, individual measurements
(Figure 14g).

8.2.4. More Information

The lack of specificity of elastically scattered light means that
particles landing at an interface are only distinguishable by

their scattering contrast. Combining scattering with fluorescence
could help distinguish species in a complex mixture or add a
distance readout – through Förster resonance energy transfer.
Polarisation-resolved measurements could also add orientational
information while acquisition of scattering at multiple colors
could add specificity to the experiment (Figure 14h).

**8.3. Conclusions**

Light scattering microscopy has recently demonstrated its major
potential as a universally applicable approach for studying
biomolecular structure and dynamics. To date, the most pow-
erful implementations of this approach have been based on
interferometric scattering detection on microscope cover glass.
Given that there is now ample evidence that label-free detection
and quantification of single biomolecules in solution is not
only possible, but can deliver additional information about the
species of interest with high measurement precision, alternative
approaches to illumination, immobilization, and detection are
likely to bring about significant improvements over the next
decade. Central to this success will be a genuine broadening
of the research base – away from a relatively small number of
highly specialized approaches towards much broader efforts
directed at the key components involved in the measurement
including and combining: detection, illumination, data analysis,
and very importantly – assays. Some of the most dramatic recent
technological advances for the life sciences have required such
interdisciplinary, and concerted efforts – and similar approaches
will be required to realize the full potential of light scattering-
based studies of biomolecules. A natural question to ask: where do these improvements end, what are the ultimate limits and how will we get there? There are two simple answers: one is incremental in all relevant aspects (detectors, light sources, sample preparation, data analysis). How powerful such an approach can be, has been demonstrated by the dramatic improvements in cryoEM over the past decade. Alternatively, we could encounter a step change – although it appears that this would necessarily require some mechanism by which either the power density, or the cross section would need to be dramatically enhanced. In principle solutions to this problem exist, how applicable they are for dynamic investigations of biomolecules remains the question.

# 9. Coherent Brightfield Microscopy

*(Yi-Teng Hsiao, Chia-Lung Hsieh*)

## 9.1. Introduction

Transmission brightfield microscopy is the earliest and arguably the simplest configuration of optical microscopy. Because of its simplicity and robustness, the brightfield microscopy has been the workhorse for histopathology for over a century (e.g., Hema-toxylin and Eosin (H&E) stained tissue samples).[305] Employing the transmission geometry, the phase contrast microscopy and Nomarski's differential interference contrast (DIC) microscopy have become the standard tool for visualization of biological cells.[306] The transmission geometry has also been widely adopted in digital holographic microscopy and quantitative phase imaging where the optical phase delay of the specimen is measured for reconstructing its organization.[247, 307, 308] Conventional transmission microscopy employs incoherent light sources (e.g., incandescent/mercury lamp and LEDs) for illumination to avoid the troubles of speckle. However, due to the low spatial coherence, it is difficult to deliver the incoherent light to the sample at a high intensity, which eventually restricts the detection sensitivity. White light illumination also experiences chromatic dispersion that often lowers the optical contrast of the image. Here, we introduce a brightfield microscopy that is equipped with a highly coherent laser for illumination, which is referred to as “coherent brightfield (COBRI) microscopy”.[88]

## 9.2. COBRI Microscopy: Detecting Forward Scattering Light Through Interference

The simplest form of COBRI microscopy is depicted in Figure 15a where a condenser lens focuses a stationary laser light onto the sample, and a microscope objective together with a tube lens form a 4F system to record the transmitted image with a digital camera. In this configuration, the illumination area has a Gaussian profile whose size is determined by the numerical aperture (NA) of the condenser lens. More uniform illumination over a defined area can be achieved by rapid scanning of a focused beam where the NA of the condenser is increased by using an objective and the beam scanning is performed by a two-axis

acousto-optic deflector (AOD) (see Figure 15b).[88] Compared with the conventional brightfield microscopy with white light illumination, COBRI microscopy offers a higher image contrast of biological cells and nanoparticles mainly because the chromatic dispersion is avoided.[88] Moreover, the laser illumination of COBRI easily provides the local light intensity required for ultrahigh speed image recording (e.g., an intensity of 10 µW µm⁻² at the sample for the acquisition rate 100 000 fps).[88]

The detected intensity of COBRI image is the superposition of the forward scattered signal of the sample and the non-scattered transmitted reference beam.

$$I_{det} = |E_t + E_s|^2 = |E_t|^2 + |E_s|^2 + 2|E_t| |E_s| \cos \theta \quad (16)$$

where $I_{det}$ is the detected intensity; $E_t$ and $E_s$ are the optical complex fields of the transmitted reference light and the forward scattered light of the sample, respectively; $\theta$ is the phase difference between $E_t$ and $E_s$. When imaging a sub-diffractional object, $\theta$ is a function the axial position of the sample relative to the focus of microscope objective as the result of Gouy phase shift, where both constructive ($\theta = 0$) and destructive ($\theta = \pi$) interference could occur (Figure 15c).[88] For small nanoparticles, $|E_s| \ll |E_t|$ and the interference term is much greater than the scattering intensity $|E_s|^2$. The interference also brings the detection into the shot-noise limited regime that is nearly immune from the detector noise.[309] More discussion on the comparisons between the interferometric and darkfield schemes can be found in other reviews.[310, 311] The COBRI contrast can be enhanced by back-pupil function engineering via inserting a dot-shaped attenuator at the Fourier plane of the sample that preferentially reduces the reference beam intensity (Figure 15b).[312] With contrast enhancement, single 10 nm AuNPs were successfully visualized and their diffusive motions are resolved with 10 nm spatial precision at a frame rate of 1000 Hz.[312]

As a common-path image-based interferometry, the COBRI microscopy shares many features with the interferometric scattering (iSCAT) microscopy, and COBRI can be considered as the transmission counterpart of iSCAT.[313] We point out that the relatively weak intensity of the reference beam in iSCAT microscopy (typically served by the reflection at the interface of supporting coverglass and the aqueous medium) readily enhances the optical contrast for detection. While the transmitted reference beam is inherently strong in COBRI, by corresponding back-pupil function engineering that attenuates the transmitted reference beam, it has been shown that COBRI microscopy offers the identical detection sensitivity as iSCAT under the same amount of illumination intensity.[312]

The key difference between COBRI and iSCAT microscopy relies on the choice of signal for detection: forward scattering in COBRI and backscattering/reflection in iSCAT. Although the detection geometry has little effect on single-particle imaging, it results in considerable differences when imaging biological cells. Figure 15d displays the COBRI and iSCAT images of a biological cell. Note that the detection sensitivity of COBRI and iSCAT are matched intentionally by adopting the suitable back-pupil function engineering in COBRI. Comparing these two images in Figure 15d, it is clear to see that the iSCAT image contains the interference rings that are absent in the
<figure><img src="image_18.png" /><figcaption>**Figure 15.** Coherent brightfield (COBRI) microscopy. a) Schematic of the simplest configuration of COBRI microscopy with a stationary widefield illumination. b) Schematic of COBRI microscopy with beam-scanning unit and contrast-enhancement unit. c) COBRI images of a virus particle at different axial positions, showing z-dependent COBRI contrast. Reproduced with permission.[88] Copyright 2017, American Chemical Society. d) Side-by-side comparison of the COBRI (left) and iSCAT (right) images of a HeLa cell cultured on coverslip. The dashed lines mark the boundaries of the nucleus and the cell body.</figcaption></figure>

COBRI image. This is because the cell surface membrane is reflective and its height is spatially varying, giving an additional heterogeneous interference pattern.[314, 315] This membrane reflection has also been observed in the early works of interference reflection microscopy (IRM) and reflection contrast microscopy (RCM),[316–318] which can be considered as the iSCAT microscopy with low-coherence light illuminations. The iSCAT signal of the live cell plasma membrane can be highly dynamic due to the spatial fluctuation of the membrane. The strong and dynamic signal of the cell membrane may complicate the imaging of other objects in the cellular environment, e.g., tracking artificial and biological nanoparticles. In contrast, the cell membrane is transparent in transmission and has negligible effect on the COBRI imaging of biological cells. Therefore, the live cell imaging is expected to be easier by COBRI microscopy in transmission (except for measuring the membrane morphology where the iSCAT microscopy in reflection is clearly more suitable).

9.3. Imaging and Tracking Single Nanoparticles by
Ultrahigh-Speed COBRI Microscopy

The COBRI microscopy without contrast enhancement readily enables direct visualization of biological nanoparticles with a diameter > 150 nm in diameter.[88] Under COBRI, single vaccinia virus particles (≈250 nm) and native intracellular vesicles can be clearly visualized (Figure 16a). The biological nanoparticles constantly move in the live cells, which undergoes the diffusive motion driven by thermal fluctuation and the directional active transportation. By using a high-speed CMOS camera as the detector, the movement of single particles can be captured with microsecond time resolution (10⁵–10⁶ frames per second). The position of the nanoparticle can be determined via single-particle localization by fitting its image with a point-spread function (PSF) function.[319, 320] Besides the scattering signal of the bionanoparticles of interest, COBRI also records the scattering from other cell structures that appears as a
<figure><img src="image_19.png" /><figcaption>**Figure 16.** Ultrahigh-speed 3D tracking of a single vaccinia virus particle on the surface of a live cell by COBRI microscopy. a) COBRI image of single vaccinia virus particles. b) Estimation and removal of cell background enables background-free imaging of the virus particle. c) Localization precision of a single virus particle is better than 3 nm in 3D. d) Lateral diffusion trajectory of a virus particle on the live cell plasma membrane, recorded at 100 000 fps. e) Reconstructed 3D trajectory of the virus particle. f) Highly transient (sub-millisecond) nano-confinements of the virus particle detected in the trajectory (highlighted in red). g) Representative trajectory segments of transient confinements of the virus particle. Adapted with permission.[88]
Copyright 2017, American Chemical Society.</figcaption></figure>

spatially heterogeneous background. This nonspecific scattering background complicates the direct visualization of the particle and lowers the precision of particle localization, posing a general challenge in detection and localization of particles by scattering-based microscopy.[319, 321] An effective remedy for the nonspecific background is to measure the background separately and remove it from the raw image by image post-processing. In the case of tracking nanoparticles in biological cells, the particle is often much more dynamic than the large biological cell structures, so the moving particle signal can be distinguished from the relatively static cell background based on their different temporal characteristics. A common strategy for background estimation is to calculate a temporal averaged image of a video that is recorded at a high speed (Figure 16b).[321, 88] More effective background estimation can be achieved by exploiting a priori knowledge of the PSF.[319] With proper background removal and signal calibration, single virus particles and cell vesicles can be tracked with a spatial precision of a few nanometers in 3D (Figure 16c).[322, 88] The strategies of fast interference image recording and processing are generally applicable to iSCAT imaging in reflection. The high-speed and accurate single-particle tracking by iSCAT has been demonstrated in live cell environments.[320]

Figure 16d displays the trajectory of a single vaccinia virus particle on the plasma membrane of a live cell, recorded at 100 000 fps with 3 nm spatial precision. The data show that the virus particle diffuses rapidly on the surface of the cell. The 3D reconstruction of the diffusion trajectory represents the morphology of the cell surface (Figure 16e). Interestingly, the ultrahigh spatiotemporal resolution captures the highly transient (sub-millisecond) confinements of the virus particle at the nano-sized zones, presumably as the result of interactions between the virus and the membrane receptors (Figure 16f,g). The detection of these short-lived events occurring at the nanoscale requires exquisite sensitivity, stable signal, and high spatial and temporal resolutions simultaneously, which are supported simultaneously by the inter-

ference scattering imaging of COBRI microscopy. Using similar methods, the native intracellular vesicles can be visualized and their bidirectional motions are resolved with microsecond time resolution and nanometer spatial precision.[322]

## 9.4. Future Perspectives

The common-path image-based interferometry, e.g., iSCAT/COBRI microscopy, exhibits excellent stability that enables sensitive detection, visualization, and tracking of very small, nano-sized particles. The reliable scattering signal facilitates ultrahigh-speed measurements that has been challenging for the fluorescence-based techniques due to the photobleaching effect.[323, 324] One of the applications is to study single-molecule dynamics by using gold nanoparticles as the scattering nanoprobes of the molecules.[311, 325, 326] It has been shown that with proper labeling, the nanoparticle introduces no artifacts to the membrane molecule, and thus the single-molecule diffusion can be resolved at the unprecedented spatiotemporal regimes.[320, 325] Using metallic nanoparticles as imaging probes has the advantages of plasmon-enhanced optical cross sections, which facilitates simultaneous scattering- and absorption-based photothermal imaging that is useful for discriminating the particle signal from the biological background.[327, 328]

The high sensitivity of iSCAT/COBRI microscopy readily supports immense applications of bioimaging beyond tracking single nanoparticles. Indeed, the recent label-free phase imaging have resolved various cell organelles with high resolutions.[329] In principle, with a higher sensitivity, iSCAT/COBRI microscopy could detect more nanoscopic objects in the cell. The critical issue is the signal-to-background ratio because the complex biological environment is expected to create considerable nonspecific background. It is recently shown that the signal-to-background ratio can be improved by exploiting the rich temporal information
encoded in a dynamic measurement.[330] Machine learning has been used to establish the complex relationships between the interference images of a biological sample and the distributions of specific molecules of interest.[181, 182, 331] The combination of iSCAT/COBRI imaging and machine learning-assisted data processing would provide the opportunity for label-free, high-speed, long-term, multiplex bioimaging.

We foresee that iSCAT/COBRI microscopy will enable exciting label-free imaging of biological cells. Because the cells have small structures compared to the optical wavelength, there is a need to improve the spatial resolution of scattering-based interference microscopy. There have been a few demonstrations of resolution enhancement by phase-sensitive interference microscopy.[332, 333] The general concept is to record the high spatial frequency signal by sequential oblique illuminations. Similar strategies can be exploited to improve the spatial resolution of iSCAT/COBRI microscopes. In addition, the spatial resolution can be improved by digital deconvolution.[95] We anticipate that the quantitative amplitude and phase retrieval by iSCAT/COBRI will soon become routine, making them the highly sensitive quantitative phase imaging (QPI) modalities. With the QPI by iSCAT/COBRI microscopy, the deconvolution can be exploited to enhance the spatial resolution, potentially reaching the super-resolution regime.

# 10. Plasmon-Enhanced Optical Ptychography and Phase Imaging

*(Brian Abbey)*

## 10.1. Introduction

The recovery of phase information from measurements of the diffracted intensity or flux of a wavefield is the basis for many types of phase contrast imaging. These techniques generally do not necessitate any staining or labeling of the sample and involve the computational reconstruction of quantitative phase information.[334, 335] Of the numerous types of label-free coherent imaging methods that have emerged, ptychography has proved to be one of the most successful. Originally developed in the context of Scanning Transmission Electron Microscopy (STEM), ptychography has already had a significant impact for X-ray,[336] electron,[337] as well as optical microscopy.[338–340]

The original concept for the ptychographic method is generally credited to Walter Hoppe, who published a series of papers on the topic in the late 1960's and early 70's.[341] Hoppe's idea was to collect a dataset comprising multiple diffraction patterns, formed by laterally translating the sample with respect to a finite illumination in discreet overlapping steps. The diffraction data then constitutes a convolution of the sample with itself, and the localized illumination function which is common to all datasets. This convolution or 'folding' of the diffraction data gave rise to the name ptychography, first used by Hegerl and Hoppe in 1972, where 'ptycho' comes from the Greek word 'to fold' ("πτυξη").[342] The lateral shifting of the position of the illumination with respect to the real-space position of the sample, has the rather profound effect of creating a phase ramp in reciprocal-space. Provided the relative location of each of the measurement points can be determined, the phase relationship between the diffraction patterns can be uniquely determined.

Unlike most optical microscopy techniques, the spatial resolution of ptychography is not limited by the properties of any image forming lenses that may be used. Instead, the spatial resolution of ptychography is determined by the Numerical Aperture (NA) subtended by the square of the modulus of the wavefield (intensity) measured at the detector plane. This means a high-resolution image can be reconstructed using low NA lenses[96] or even using no optics at all.

It has also been reported that by exploiting the high degree of data 'redundancy' in ptychography or via combination with aperture scanning it is possible to obtain a 'super-resolution' image beyond the diffraction limit.[86] There are three mechanisms applicable to ptychography which contribute to improving the spatial resolution beyond the numerical aperture of the detected diffraction signal. First the "synthetic aperture" effect in which data is collected from the specimen using a plane wave incident at different angles.[343] Data from this different range of scattering angles can be combined to form a larger "synthetic" diffraction pattern on the detector which can be inverted to form a super-resolved image. Second, "analytic continuation" in which the measured diffraction signal is treated as the convolution of the spatial frequency spectrum of the specimen convolved with the Fourier Transform of the exit pupil of the optical system aperture by the detector.[344] Convolution ensures that the recorded diffraction data contains information from the specimen's entire frequency spectrum enabling extrapolation of the diffraction patterns beyond the measured data. Finally, 'dithering': sub-pixel lateral shifting of the specimen by a non-integer number of pixels can also contribute to an enhanced resolution (although this may be more accurately described as de-aliasing as opposed to super-resolution).[345] Achieving super-resolution in ptychography depends on successfully exploiting one or more of these effects. Another advantage afforded by optical ptychography is contrast. The ability to generate label-free, high-contrast, quantitative amplitude and phase information is attractive for bioimaging. A key feature differentiating ptychography from other phase contrast techniques is that the complex probe function is typically computationally recovered in a simultaneous operation along with the sample refractive index.[346] This avoids errors which can be introduced via an imperfect knowledge or measurement of the incident illumination and provides a reference against which the quantitative optical properties of the sample can be retrieved.

Recently there has been significant interest in further improving the sensitivity of optical phase microscopy by exploiting plasmonic nanostructures.[347] The combination of phase imaging and plasmon-enhanced sensing has the potential to deliver exceptional sensitivity for weakly interacting specimens. Phase imaging setups incorporating plasmon-enhancement include interferometry (Figure 17A-E),[348] holography,[349] Differential Interference Contrast (DIC),[350] and ptychography (Figure 17F-G).[87] Analysis of coherent scattering has also been used as a means of visualising the complex optical plasmon fields generated by metallic nanostructures using near-field scanning microscopy. For example, Hillenbrand et al. were able to directly image the phase and amplitude associated with resonantly induced plasmon oscillation to a spatial resolution of 30 nm using a carbon nanotube as the optical probe.[334] In the case of Plasmon-Enhanced ptychography (PE-ptychography) enhancement of both the amplitude and phase for histological
<figure><img src="image_20.png" /><figcaption>**Figure 17.** Experimental setups and mechanism for interferometric plasmonic microscopy (a–e) and plasmon-enhanced ptychography (f–g). a) Schematic showing the interferometric plasmonic microscopy (iPM) setup. b) iPM real-space image showing a 40 nm Ag nanoparticle and c) in reciprocal space. e) Diagram showing incident angle ($\theta_i$) and resonance angle ($\theta_{SPR}$). Inc (incident beam); Obj (objective); BFP (back-focal-plane); LR (leakage radiation); Ref (reflective beam); TL (tube lenses). Reproduced with permission.[350] Copyright 2019 American Chemical Society. f) Experimental set-up for PE-ptychography. g) Schematic of the evanescent electromagnetic field generated by SPPs. EM, electromagnetic; TM, transverse magnetic; TE, transverse electric. Reproduced with permission.[87] Copyright 2021 Springer Nature.</figcaption></figure>

tissue sections was demonstrated (Figure 18a-b). Typically, the variation of amplitude and phase for biological samples is quite linear over the visible range. However, the introduction of plasmon-mediated near-field interactions induces significant and often contrasting variations in the complex Refractive Index (RI) as a function of wavelength (Figure 18c-f).

## 10.2. Current Status and Future Prospects

The incorporation of nanostructures into phase imaging has already been shown to deliver superior performance in terms of contrast. However, for most plasmon-enhanced phase imaging, sub-diffraction limited super-resolution has yet to be clearly demonstrated. In the case of polarization-controlled holographic imaging of nanoantennas[349] the authors resolve the amplitude signal arising from individual nanoantenna's to a peak width of approximately 0.3 µm. This was using a source comprising a tungsten-halogen lamp filtered to produce a central wavelength of 600 nm with a full-width-at-half-maximum (FWHM) of 50 nm. Other proposals for achieving super-resolution, plasmon-enhanced phase imaging include the recovery of the high frequency components of the scattered signal beyond the NA of the imaging system, which are modulated by sur-

face plasmons.[350] This is possible using a variety of different image reconstruction techniques including synthetic aperture microscopy[348] and ptychography.[86]

To date, super-resolution using PE-ptychography has yet to be experimentally demonstrated whilst other the label-free approaches have thus far only achieved relatively modest (up to 25%) improvement in spatial resolution beyond the diffraction limit.[349, 350] However, these ideas are still relatively new and have not yet had time to fully realize their potential for label-free, super-resolution bioimaging. Other more practical issues to consider are ease of use and cost of fabrication. For instance, some of the methods proposed for plasmon-enhanced phase imaging involve complex aperiodic patterning or expertise in image reconstruction and analysis. These issues, in the long term, may prove challenging for the wide adoption of these techniques by the optical microscopy community.

## 10.3. Next Steps in Development

The advantages of using plasmonics in terms of sensitivity are further augmented by access to quantitative phase information. Critically, this information can help to improve specificity to different types of analytes or cells which is a key feature
<figure><img src="image_21.png" /><figcaption>**Figure 18.** Plasmon-enhanced images of ocular nerve tissue and simulated amplitude and phase contrast predicted for different thicknesses of carbon. a) Ptychography without plasmon enhancement and b) PE-ptychography. The graphs represent line outs through the same 70 nm thick edge feature on the histological tissue section, with significantly enhanced contrast observed in both the amplitude and phase when using PE-ptychography. Note that the spatial resolution for both ptychography and PE-ptychography, based on lineouts of the edge of the tissue, are equal to 719 nm ± 5 nm (λ = 660 nm). The two vertical lines indicate the two different wavelengths (λ) at which ptychography data was collected during the experiment. c) Amplitude and d) phase variation as a function of wavelength without the plasmonic device, e) amplitude and f) phase variation as a function of wavelength with the plasmonic device in place for four different carbon thicknesses indicated in the top right-hand corner of c). Reproduced with permission.[87] Copyright 2021 Springer Nature.</figcaption></figure>

required for diagnostic applications. Very few (if any) of these concepts, however, have progressed to the point at which they could be applied within a clinical setting although they have shown potential for significantly improving the sensitivity and resolution of conventional phase imaging. A key feature of these techniques is that the interaction between surface plasmons and biological specimens fundamentally alters the complex sample Exit Surface Wave (ESW). This creates challenges for quantitatively interpreting the reconstructed phase in terms of the local properties of biological samples (e.g. dry density and dry mass of cells). Whereas extracting useful biological information from conventional quantitative phase contrast measurements is comparatively straightforward, caution must be applied when making the same types of claims in relation to plasmon-enhanced phase imaging. The RI information measured using these approaches is the result of a modification of the plasmon-resonance structure generated by the active nanostructures and the local complex dielectric constant of the sample. The latter is often spatially varying on length scales which may be comparable to the spatial resolution of the measurement or even the to the size of the plasmonically active nanostructures themselves. Deconvolution of the sample properties from the near-field response of the plasmonic device will require significant work to understand and model the properties of both the device and sample. In the case of biological cells and tissues it is not yet clear whether it is even feasible to be able to replicate the complexity and local heterogeneity of such samples in simulation. Although, there have been attempts in the literature to construct such models which may potentially be used to represent the 'average' properties of the sample.[351] Hence, significant research is required to enable plasmon-enhanced phase imaging to achieve comparable levels of quantitative reliability in terms of bioimaging as conventional phase contrast microscopy techniques such as interferometry or holography.

Whilst label-free plasmon-enhanced phase imaging offer many opportunities in terms of flexibility, applicability, and ease-of-use, there is considerable scope to improve specificity still further. This is particularly true in applications where label-free bioimaging is being developed as a potential adjunct or even alternative to molecular labeling. An obvious next step to improving specificity is to combine plasmon-enhanced phase contrast microscopy with hyperspectral imaging techniques. Since the variation in the complex RI of plasmonic devices is typically highly non-linear with wavelength (Figure 18c-f), different sample components have vastly different optical contrast properties depending on the incident wavelength. Hence, merging these two approaches could deliver exceptional performance in differentiating (sub-)cellular components and distinguishing them from the surrounding host environment.

However, the generation of large spatially and spectrally resolved datasets creates new challenges in terms of analysis. Deep learning has already been demonstrated as a potential tool for optimizing the design and predicting the output of plasmonic nanostructures.[352] It has also been proposed as a technique for analyzing the plasmonic resonance output of these devices.[353] Deep-learning algorithms have already been applied successfully for a variety of biosensing and bioimaging applications. For example, a machine learning approach was recently used to optimize the design and fabrication of plasmonic nanostructures for mobile broadband RI sensing (Figure 19).[354] Given the complexity and volume of data that could be generated by plasmon-enhanced phase imaging of biological specimens the application of machine learning algorithms could revolutionize the analysis of label-free plasmon-resonance data. It may enable the extraction and generation of multiplexed images from a single dataset and could even be used to generate a 'fingerprint' or 'barcode' for certain classes of cell or disease state.[355] Whilst the implementation and integration of deep learning approaches

<caption>**Figure 18.** Plasmon-enhanced images of ocular nerve tissue and simulated amplitude and phase contrast predicted for different thicknesses of carbon. a) Ptychography without plasmon enhancement and b) PE-ptychography. The graphs represent line outs through the same 70 nm thick edge feature on the histological tissue section, with significantly enhanced contrast observed in both the amplitude and phase when using PE-ptychography. Note that the spatial resolution for both ptychography and PE-ptychography, based on lineouts of the edge of the tissue, are equal to 719 nm ± 5 nm (λ = 660 nm). The two vertical lines indicate the two different wavelengths (λ) at which ptychography data was collected during the experiment. c) Amplitude and d) phase variation as a function of wavelength without the plasmonic device, e) amplitude and f) phase variation as a function of wavelength with the plasmonic device in place for four different carbon thicknesses indicated in the top right-hand corner of c). Reproduced with permission.[87] Copyright 2021 Springer Nature.</caption>
<figure><img src="image_22.png" /><figcaption>**Figure 19.** Schematic illustration of the machine learning framework. Reproduced with permission.[354] Copyright 2017 American Chemical Society.</figcaption></figure>

into plasmon-enhanced bioimaging is currently technically feasible, caution must be applied based on the very limited (or in many cases non-existent) availability of high-quality training data. For example, although it appears feasible to train a neural network to quantitatively analyze PE-ptychography data using a combination of plasmon-resonance data plus high-quality simulations, at the time of writing, such ideas have yet to be explored.

## 10.4. Conclusions

The addition of quantitative phase information to plasmonic microscopy opens up many new and exciting avenues of research for label-free bioimaging. In terms of 'real-world' biological samples, however, most of the published techniques are still in the early stages of development. As new approaches to large-scale, high-quality nanofabrication become increasingly accessible[356] plasmonic devices are expected to become increasingly common in bioimaging. Exploiting the specific interaction of the local sample dielectric constant with plasmonic nanostructures in combination with phase imaging could add a much higher degree of specificity, facilitating multiplexed imaging and enabling specific (sub-) cellular features to be highlighted. As larger, higher-quality datasets become available there are also opportunities to exploit deep learning approaches which offer the promise of maximizing the amount of information that can be extracted from plasmon-enhanced phase imaging data. In terms of achieving super-resolution, several of the techniques discussed have already achieved spatial resolutions just beyond the diffraction limit. In the case of plasmon-enhanced ptychography, there are several possible pathways to super-resolution although the reconstruction of higher order plasmon-mediated diffraction data, be-

yond the numerical aperture, is likely the most promising avenue for future development.

# 11. Circular Intensity Differential Scattering Microscopy

(Alberto Diaspro*, Aymeric LeGratiet, Paolo Bianchini)

(Dedicated to the memory of Ignacio Tinoco Jr. (1930-2016) and Gary C. Salzman (1942-2020)).

## 11.1. Introduction

The transition of optical microscopy to optical nanoscopy allows to super-resolve the position of objects and, more interestingly, of biological molecules to a spatial resolution better than the one imposed by the wavelength of light. It is not related to violating physical laws but of implementing the smart concepts analyzed by Toraldo di Francia and Luksosz.[357] namely: adding information to the light detection process and reducing the field of view.

The coupling with fluorescence labeling of the object to be visualized permits to spatially resolve details of intricate extended cellular substructures. Recently, technological advances in sensing photons allowed to extend the capabilities in the temporal domain adding spectroscopic fluorescence features, including lifetime and polarization.[358] However, fluorescence realized by specific labeling as a contrast mechanism has shown some limitations also when boosted by the nanoscale performances. Among them, single molecule imaging poses a problem related to possible effects on crowding in terms of re-organization of important biological macromolecules like chromatin in the cell[359] nucleus and their labeling, as also reported by means of a comparative approach.[360] In this scenario, methods that use a label-free
approach as a contrast mechanism go beyond the trend to become highly relevant for studying structure and function relationships in biological systems.[149] This fact also has a considerable impact on the development of new diagnostic instrumentation in medicine like those using polarization as mechanism of contrast.[361, 362] It is a fact that the variety of advanced Stokes-Mueller approaches and the growing aptitude for multimodal information gathering combined with artificial intelligence can be an additional strength for applications in biomedical and clinical research.[363]

Circular Intensity Differential Scattering (CIDS) is a label-free optical biophysical method originated as a differential measurement of the circular right and left polarized light scattered by a specimen outside the absorption bands instead of considering its differential absorption as in the case of circular dichroism. CIDS can be also seen as one of the 16 elements of the Mueller matrix used to generate multiparameter label-free optical images in microscopy. This approach was proven to be a gold standard label-free technique to study the molecular conformation of important biological macromolecules like chromatin and to enable rapid identification of a variety of viruses of clinical significance. In early works, it was demonstrated the potential of studying long-range chiral organization on a scale down to 1/10th–1/20th of the illumination wavelength. This hands the approach to label-free nanoscale optical microscopy. In this review, we discuss methods and technologies used for measuring this signal coming from complex macro-molecules ordering.

## 11.2. Circular Intensity Differential Scattering (CIDS)

CIDS microscopy has its foundation in the development of the theory of differential imaging as extension of the experiments carried out on the interaction of circularly polarized with structures whose size is arbitrary with respect to the wavelength of the illumination.[364]

More specifically, the circularly polarized light was used to study biological macromolecules, large oriented biological helices, in terms of circular dichroism, CD.[365] Circular dichroism is the difference in the extinction coefficient of matter imaged by left and right circularly polarized radiation. When investigating specimens containing large biomolecular motifs, CD signals produced an unexpected signature outside the absorption bands.[366] This signature was proven to be due to differential scattering cross section.[364] The spatial distribution of the angular scattered light which considers the preferential scattering of left and right circularly polarized illumination by the specimen at a given direction is defined as:

$$ \text{CIDS}(\theta) = \frac{I_{\perp}(\theta) - I_{\parallel}(\theta)}{I_{\perp}(\theta) + I_{\parallel}(\theta)} \qquad (17) $$

where $I_{\perp}$ are the detected intensities for the left and right circular polarization states at a certain scattering angle ($\theta$), respectively.[364] A paradigmatic example was early given by CIDS measurements of a suspension of the helical sperm head of the Mediterranean octopus *Eledone cirrhosa*.[367] Its left-handed helix approximately 43 µm long, with a pitch and radius of 1 µm and 0.65 µm, respectively, demonstrated the availability of strong

signals in the wavelength regions above 300 nm outside the absorption bands of the nucleic acid acids and proteins components of the sperm head.[368] Theoretical analysis of the optical models have shown that CIDS signatures are most sensitive to the helical parameters.[364, 369, 370, 89] Both theory and experiments lead to the observation that the differential scattering cross section is a function of higher chiral order of important biopolymers like chromatin-DNA.

## 11.3. Discussion

The CIDS signature was initially measured on solutions demonstrating the ability of discriminating viral[371] and bacterial[372] species opening an interesting window of medical applications. Moreover, the results obtained on isolated chromatin in solution at different level of organization,[373] induced by modulating the salt concentration, pointed out to the charming possibility of monitoring chromatin compartmentalization in the cell nucleus.[374]

The development of a microscope was pushed by the availability of the photo elastic modulator, as device to control polarization states of the illumination and of the detected photons.[375] The CIDS signal is also part of a multiparameter analysis that can be done using the Mueller matrix approach. In this approach, modulating four states of polarization one can produce sixteen data set that can be transformed in label-free images when implemented in the optical pathway of an optical microscope.[375]

To take advantage of a beam scanning architecture, a fast generation of the polarization states to obtain a good signal-to-noise ratio at the pixel-dwell time rate, typically in the µs range, one can also exploit the simultaneous emission of orthogonal polarization states from a Zeeman effect laser providing alternate polarization states without the need for further active components.[376] At the very same time allowing compactness and simplicity of construction that are key elements for a more effective dissemination of label-free techniques and their use in biology and medicine. Another step to facilitate the interpretation of the multiple parameter data encoded in the polarization-based signals is given by the combination of Mueller matrix microscopy with the phasor approach.[377] The full matrix is converted into four Stokes vector[378] images and then a Fourier transform algorithm is applied to each of these images.[379]

**Figure 20a** shows an effective implementation of a multiparameter CIDS microscope also endowed of fluorescence capabilities. The illumination is implemented using a Ti-Sapphire microscope able to prime two-photon excitation fluorescence and at the very same time to generate a CIDS signal using a photoelastic modulator operating at the operating frequency $\omega$ and a lock-in amplifier selecting the 1st and 2nd harmonic signals at $\omega$ and $2\omega$. These signals plus the DC signal allow to extract Mueller matrix elements intensities including CIDS, the 4th element of the 1st row of the matrix.[374, 375] In case of a viral sample, for example the coronavirus sars-cov2 as sketched in Figure 20b, one can predict a potential in discriminating among spherical droplets containing spherical particles endowed of spikes.[378] This is in tune with the rapid microbial detection early demonstrated on flu viruses and other infections.[371] The phasor representation can be used to allow medical doctors and clinicians to visualize
<figure><img src="image_23.png" /><figcaption>**Figure 20.** a) Block diagram of the CIDS scanning microscope. The red and green arrows correspond to the transmitted polarimetric and to the reflected fluorescence path, respectively.[374] b) An example of theoretical results about the presence of a spike-virus in a spherical droplet with size comparable to sars-cov2.[378] c) Use of the Phasor analysis to distinguish species on the basis of CIDS signature[380]; d) Normalized CIDS (blu) image of an isolated HEK nucleus after extraction compared with fluorescence intensity (green) signature based on DNA labeling.[374] Labels: Ti:Sa: Titanium-Sapphire coherent laser source tuned at 740 nm to prime both CIDS and two-photon excitation fluorescence. SU: Scanning Unit. PEM: Photoelastic Modulator at 50 kHz resonant frequency. Obj1: Microscope objective to image the sample. Obj2: Microscope objective used as a condenser to collect the transmitted light. GT: Glan-Taylor prism. APD1 and APD2: Avalanche Photodiode for + 45° and -45° polarization detection after the GT. LA: Lock-in Amplifiers. LA#1 and LA#2: input channels of the LA, locked at 50 kHz from the reference signal of the PEM. The two APDs are both connected to each channel of the LA and also directly to the control unit. The terms “I +” and “I −” are the detected intensities + 45° and -45° polarization projections after the GT, respectively. a,d) Adapted with permission.[374] Copyright 2018, The Authors, published by Optica Publishing Group. b) Adapted with permission.[378] Copyright 2021, The Optical Society.</figcaption></figure>

the possible presence of a classified population in a friendly and
effective way.[380] Figure 20c shows an example based on a pop-
ulation classification. Figure 20d shows the label-free ability of
detecting a different compaction of chromatin in the cell nucleus
using “classical” fluorescence DNA signature as benchmark.[374]

**11.4. Conclusion**

Studies of nuclear architecture reveal that the dynamic proper-
ties of nucleic acids and proteins in the nucleus are critical for
their function. In the case of labeling with fluorescent molecules
of different sizes, despite the essential spectroscopic and mor-
phological information that one can collect at an unprecedented
available temporal and spatial resolution, one has two aspects
that are worth to be considered, namely: 1) the crowded condition
of high-density biopolymers increases producing an expected
effect in terms of mobility and organization; 2) not always one
can fluorescently label a sample, especially within a clinical
perspective. CIDS can sense the conformation of biological
macromolecules and identification of viruses of clinical signif-

icance at the nanoscale offering a map that is comparable to
the one built using super resolved localization of fluorescent
tags. The coupling with artificial intelligence algorithms and
super-resolved fluorescence microscopy approaches offers the
chance of performing an interesting training of supervised
machine learning programs towards the transformation of label
free images into super resolved ones, for example using a CNN
(Convolutional Neural Network) operating on CIDS and fluores-
cence maps stemming from the very same sample.[182, 381] Since
CIDS and other polarization related parameters demonstrated
the ability to provide information related to the molecular level,
one can predict that polarization-based contrast mechanisms will
soon play an important role in cell and molecular biophysics.
Moreover, CIDS applications range from those instruments
using synchrotron radiation and ordinary sources[368] from virus
identification to organizational motifs of chromatin in the cell
nucleus allowing to access structural information from atoms to
surfaces, from single cells to tissues and organs in a label-free
way.[382] Other elements of the Mueller matrix offer the very same
chance of CIDS pending a model establishing the relationship
between label-free and fluorescence imaging.[370, 373, 374, 89]