Input: What are the two ground states observed for indeno[1,2-a]fluorene on NaCl surfaces?

Context:
Paper Info

Title: Bistability between π-diradical open-shell and closed-shell states in indeno[1,2-a]fluorene
Publish Date: Unkown
Author List: Shantanu Mishra (from IBM Research Europe -Zurich), Manuel Vilas-Varela (from Department of Organic Chemistry, Center for Research in Biological Chemistry and Molecular Materials (CiQUS), University of Santiago de Compostela), Leonard-Alexander Lieske (from IBM Research Europe -Zurich), Ricardo Ortiz (from Donostia International Physics Center (DIPC)), Igor Rončević (from Department of Chemistry, University of Oxford), Florian Albrecht (from IBM Research Europe -Zurich), Diego Peña (from Department of Organic Chemistry, Center for Research in Biological Chemistry and Molecular Materials (CiQUS), University of Santiago de Compostela), Leo Gross (from IBM Research Europe -Zurich)

Figure

Fig. 1 | Non-benzenoid non-alternant polycyclic conjugated hydrocarbons.a, Classical nonbenzenoid non-alternant polycyclic conjugated hydrocarbons: pentalene, azulene and heptalene.b, Generation of indacenes and indenoindenes through benzinterposition and benzannelation of pentalene, respectively.Gray filled rings represent Clar sextets.c, Closed-shell Kekulé (left) and openshell non-Kekulé (right) resonance structures of QDMs.Note that meta-QDM is a non-Kekulé molecule.All indenofluorene isomers, being derived through benzannelation of indacenes, contain a central QDM moiety.d, Closed-shell Kekulé (top) and open-shell non-Kekulé (bottom) resonance structures of indenofluorenes.Compared to their closed-shell structures, 1 and 5 gain two Clar sextets in the openshell structure, while 2-4 gain only one Clar sextet in the open-shell structure.Colored bonds in d highlight the ortho-and para-QDM moieties in the two closed-shell Kekulé structures of 5. e, Scheme of on-surface generation of 5 by voltage pulse-induced dehydrogenation of 6 (C20H14).Structures 7 and 8 represent the two monoradical species (C20H13).
Fig. 2 | Characterization of open-shell indeno[1,2-a]fluorene on bilayer NaCl/Au(111).a, DFTcalculated wave functions of the frontier orbitals of 5OS in the triplet configuration for the spin up (occupied) level (isovalue: 0.002 e -Å -3 ).Blue and red colors represent opposite phases of the wave function.b, Corresponding DFT-calculated spin density of 5OS (isovalue: 0.01 e -Å -3).Blue and orange colors represent spin up and spin down densities, respectively.c, Probability density of the SOMOs of 5OS (isovalue: 0.001 e -Å -3 ).d, DFT-calculated bond lengths of 5OS.e, Constant-height I(V) spectra acquired on a species of 5 assigned as 5OS, along with the corresponding dI/dV(V) spectra.Open feedback parameters: V = -2 V, I = 0.17 pA (negative bias side) and V = 2 V, I = 0.17 pA (positive bias side).Acquisition position of the spectra is shown in Supplementary Fig.7.f, Scheme of many-body transitions associated to the measured ionic resonances of 5OS.Also shown are STM images of assigned 5OS at biases where the corresponding transitions become accessible.Scanning parameters: I = 0.3 pA (V = -1.2V and -1.5 V) and 0.2 pA (V = 1.3 V and 1.6 V). g, Laplace-filtered AFM image of assigned 5OS.STM set point: V = 0.2 V, I = 0.5 pA on bilayer NaCl, Δz = -0.3Å.The tip-height offset Δz for each panel is provided with respect to the STM setpoint, and positive (negative) values of Δz denote tip approach (retraction) from the STM setpoint.f and g show the same molecule at the same adsorption site, which is next to a trilayer NaCl island.The bright and dark features in the trilayer NaCl island in g correspond to Cl -and Na + ions, respectively.Scale bars: 10 Å (f) and 5 Å (g).
Fig. 3 | Characterization of closed-shell indeno[1,2-a]fluorene on bilayer NaCl/Au(111).a, DFTcalculated wave functions of the frontier orbitals of closed-shell 5 0 (isovalue: 0.002 e -Å -3 ).The wave functions shown here are calculated for the 5para geometry.b, DFT-calculated bond lengths of 5ortho (top) and 5para (bottom).c, Constant-height I(V) spectra acquired on a species of 5 assigned as 5para, along with the corresponding dI/dV(V) spectra.Open feedback parameters: V = -2 V, I = 0.15 pA (negative bias side) and V = 2.2 V, I = 0.15 pA (positive bias side).Acquisition position of the spectra is shown in Supplementary Fig. 7. d, Scheme of many-body transitions associated to the measured ionic resonances of 5para.Also shown are STM images of assigned 5para at biases where the corresponding transitions become accessible.Scanning parameters: I = 0.15 pA (V = -1.5 V) and 0.2 pA (V = 1.7 V). e, Laplace-filtered AFM image of assigned 5para.STM set point: V = 0.2 V, I = 0.5 pA on bilayer NaCl, Δz = -0.7 Å. f, Selected bonds labeled for highlighting bond order differences between 5para and 5ortho.For the bond pairs a/b, c/d and e/f, the bonds labeled in bold exhibit a higher bond order than their neighboring labeled bonds in 5para.g, Laplace-filtered AFM images of 5 on bilayer NaCl/Cu(111) showing switching between 5OS and 5para as the molecule changes its adsorption position.The faint protrusion adjacent to 5 is a defect that stabilizes the adsorption of 5. STM set point: V = 0.2 V, I = 0.5 pA on bilayer NaCl, Δz = -0.3Å. STM and STS data in c and d are acquired on the same species, while the AFM data in e is acquired on a different species.Scale bars: 10 Å (d) and 5 Å (e,g).
NMR (300 MHz, CDCl3) δ: 7.51 (m, 2H), 7.40 -7.28 (m, 5H), 7.27 -7.20 (m, 2H), 7.13 (d, J = 7.7 Hz, 1H), 2.07 (s, 3H), 1.77 (s, 3H) ppm. 13C NMR-DEPT (75 MHz, CDCl3, 1:1 mixture of atropisomers) δ: 141.2 (C), 141.1 (C), 140.0 (C), 139.4 (2C), 137.5 (C), 137.4 (C), 136.0 (3C), 134.8 (C), 134.5 (C), 134.1 (C), 134.0 (C), 133.7 (C), 133.6 (C), 131.6 (CH), 131.2 (CH), 131.1 (CH), 130.7 (CH), 129.8 (CH), 129.7 (CH), 129.5 (CH), 129.4 (CH), 129.0 (CH), 128.9 (CH), 128.7 (2CH), 128.6 (2CH), 127.2 (CH), 127.1 (CH), 127.0 (CH), 126.9 (CH), 126.7 (CH), 126.6 (CH), 20.6 (CH3), 20.5 (CH3), 17.7 (CH3), 17.5 (CH3) ppm.MS (APCI) m/z (%): 327 (M+1, 100).HRMS: C20H16Cl2; calculated: 327.0702, found: 327.0709.
NMR (500 MHz, CDCl3) δ: 7.93 (d, J = 7.6 Hz, 1H), 7.85 (d, J = 7.5 Hz, 1H), 7.78 (d, J = 7.7 Hz, 1H), 7.65 (d, J = 7.4 Hz, 1H), 7.61 (d, J = 7.5 Hz, 1H), 7.59 (d, J = 7.7 Hz, 1H), 7.47 (ddd, J = 8.4, 7.2, 1.1 Hz, 1H), 7.42 (dd, J = 8.1, 7.0 Hz, 1H), 7.35 (m, 2H), 4.22 (s, 3H), 4.02 (s, 3H).ppm. 13C NMR-DEPT (125 MHz, CDCl3) δ: 144.1 (C), 143.3 (C), 142.3 (C), 141.9 (C), 141.8 (C), 141.2 (C), 138.2 (C), 136.5 (C), 127.0 (CH), 126.9 (CH), 126.7 (CH), 126.6 (CH), 125.3 (CH), 125.2 (CH), 123.6 (CH), 122.2 (CH), 119.9 (CH), 118.4 (CH), 37.4 (CH2), 36.3 (CH2).ppm.MS (APCI) m/z (%): 254 (M+, 88).HRMS: C20H14; calculated: 254.1090, found: 254.1090.

abstract

Indenofluorenes are non-benzenoid conjugated hydrocarbons that have received great interest owing to their unusual electronic structure and potential applications in nonlinear optics and photovoltaics. Here, we report the generation of unsubstituted indeno[1,2-a]fluorene, the final and yet unreported parent indenofluorene regioisomer, on various surfaces by cleavage of two C-H bonds in 7,12-dihydro indeno[1,2-a]fluorene through voltage pulses applied by the tip of a combined scanning tunneling microscope and atomic force microscope.
On bilayer NaCl on Au(111), indeno[1,2a]fluorene is in the neutral charge state, while it exhibits charge bistability between neutral and anionic states on the lower work function surfaces of bilayer NaCl on Ag(111) and Cu(111). In the neutral state, indeno[1,2-a]fluorene exhibits either of two ground states: an open-shell π-diradical state, predicted to be a triplet by density functional and multireference many-body perturbation theory calculations, or a closedshell state with a para-quinodimethane moiety in the as-indacene core.
Switching between open-and closed-shell states of a single molecule is observed by changing its adsorption site on NaCl. The inclusion of non-benzenoid carbocyclic rings is a viable route to tune the physicochemical properties of polycyclic conjugated hydrocarbons (PCHs) . Non-benzenoid polycycles may lead to local changes in strain, conjugation, aromaticity, and, relevant to the context of the present work, induce an open-shell ground state of the corresponding PCHs .
Many nonbenzenoid PCHs are also non-alternant, where the presence of odd-membered polycycles breaks the bipartite symmetry of the molecular network . Figure shows classical examples of non-benzenoid non-alternant PCHs, namely, pentalene, azulene and heptalene. Whereas azulene is a stable PCH exhibiting Hückel aromaticity ([4n+2] π-electrons, n = 2), pentalene and heptalene are unstable Hückel antiaromatic compounds with [4n] π-electrons, n = 2 (pentalene) and n = 3 (heptalene).
Benzinterposition of pentalene generates indacenes, consisting of two isomers s-indacene and as-indacene (Fig. ). Apart from being antiaromatic, indacenes also contain proaromatic quinodimethane (QDM) moieties (Fig. ) , which endows them with potential open-shell character. While the parent s-indacene and asindacene have never been isolated, stable derivatives of s-indacene bearing bulky substituents have been synthesized .
A feasible strategy to isolate congeners of otherwise unstable non-benzenoid non-alternant PCHs is through fusion of benzenoid rings at the ends of the π-system, that is, benzannelation. For example, while the parent pentalene is unstable, the benzannelated congener indeno[2,1-a]indene is stable under ambient conditions (Fig. ) .
However, the position of benzannelation is crucial for stability: although indeno[2,1a]indene is stable, its regioisomer indeno[1,2-a]indene (Fig. ) oxidizes under ambient conditions . Similarly, benzannelation of indacenes gives rise to the family of PCHs known as indenofluorenes (Fig. ), which constitute the topic of the present work.
Depending on the benzannelation position and the indacene core, five regioisomers can be constructed, namely, indeno [ Practical interest in indenofluorenes stems from their low frontier orbital gap and excellent electrochemical characteristics that render them as useful components in organic electronic devices .
The potential open-shell character of indenofluorenes has led to several theoretical studies on their use as non-linear optical materials and as candidates for singlet fission in organic photovoltaics . Recent theoretical work has also shown that indenofluorene-based ladder polymers may exhibit fractionalized excitations.
Fundamentally, indenofluorenes represent model systems to study the interplay between aromaticity and magnetism at the molecular scale . Motivated by many of these prospects, the last decade has witnessed intensive synthetic efforts toward the realization of indenofluorenes. Derivatives of 1-4 have been realized in solution , while 1-3 have also been synthesized on surfaces and characterized using scanning tunneling microscopy (STM) and atomic force microscopy (AFM), which provide information on molecular orbital densities , molecular structure and oxidation state .
With regards to the open-shell character of indenofluorenes, 2-4 are theoretically and experimentally interpreted to be closed-shell, while calculations indicate that 1 and 5 should exhibit open-shell ground states . Bulk characterization of mesitylsubstituted 1, including X-ray crystallography, temperature-dependent NMR, and electron spin resonance spectroscopy, provided indications of its open-shell ground state .
Electronic characterization of 1 on Au(111) surface using scanning tunneling spectroscopy (STS) revealed a low electronic gap of 0.4 eV (ref. ). However, no experimental proof of an openshell ground state of 1 on Au(111), such as detection of singly occupied molecular orbitals (SOMOs) or spin excitations and correlations due to unpaired electrons , was shown.
In this work, we report the generation and characterization of unsubstituted 5. Our research is motivated by theoretical calculations that indicate 5 to exhibit the largest diradical character among all indenofluorene isomers . The same calculations also predict that 5 should possess a triplet ground state.
Therefore, 5 would qualify as a Kekulé triplet, of which only a handful of examples exist . However, definitive synthesis of 5 has never been reported so far. Previously, Dressler et al. reported transient isolation of mesityl-substituted 5, where it decomposed both in the solution and in solid state , and only the structural proof of the corresponding dianion was obtained.
On-surface generation of a derivative of 5, starting from truxene as a precursor, was recently reported . STM data on this compound, containing the indeno[1,2-a]fluorene moiety as part of a larger PCH, was interpreted to indicate its open-shell ground state. However, the results did not imply the ground state of unsubstituted 5. Here, we show that on insulating surfaces 5 can exhibit either of two ground states: an open-shell or a closed-shell.
We infer the existence of these two ground states based on high-resolution AFM imaging with bond-order discrimination and STM imaging of molecular orbital densities . AFM imaging reveals molecules with two different geometries. Characteristic bond-order differences in the two geometries concur with the geometry of either an open-or a closed-shell state.
Concurrently, STM images at ionic resonances show molecular orbital densities corresponding to SOMOs for the open-shell geometry, but orbital densities of the highest occupied molecular orbital (HOMO) and lowest unoccupied molecular orbital (LUMO) for the closed-shell geometry. Our experimental results are in good agreement with density functional theory (DFT) and multireference perturbation theory calculations.
Finally, we observe switching between open-and closed-shell states of a single molecule by changing its adsorption site on the surface. Synthetic strategy toward indeno[1,2-a]fluorene. The generation of 5 relies on the solution-phase synthesis of the precursor 7,12-dihydro indeno[1,2-a]fluorene (6). Details on synthesis and characterization of 6 are reported in Supplementary Figs.
. Single molecules of 6 are deposited on coinage metal (Au(111), Ag(111) and Cu(111)) or insulator surfaces. In our work, insulating surfaces correspond to two monolayer-thick (denoted as bilayer) NaCl on coinage metal surfaces. Voltage pulses ranging between 4-6 V are applied by the tip of a combined STM/AFM system, which result in cleavage of one C-H bond at each of the pentagonal apices of 6, thereby leading to the generation of 5 (Fig. ).
In the main text, we focus on the generation and characterization of 5 on insulating surfaces. Generation and characterization of 5 on coinage metal surfaces is shown in Supplementary Fig. . ). Blue and orange colors represent spin up and spin down densities, respectively. c, Probability density of the SOMOs of 5OS (isovalue: 0.001 e -Å -3 ).
d, DFT-calculated bond lengths of 5OS. e, Constant-height I(V) spectra acquired on a species of 5 assigned as 5OS, along with the corresponding dI/dV(V) spectra. Open feedback parameters: V = -2 V, I = 0.17 pA (negative bias side) and V = 2 V, I = 0.17 pA (positive bias side). Acquisition position of the spectra is shown in Supplementary Fig. . f, Scheme of many-body transitions associated to the measured ionic resonances of 5OS.
Also shown are STM images of assigned 5OS at biases where the corresponding transitions become accessible. Scanning parameters: I = 0.3 pA (V = -1.2 V and -1.5 V) and 0.2 pA (V = 1.3 V and 1.6 V). g, Laplace-filtered AFM image of assigned 5OS. STM set point: V = 0.2 V, I = 0.5 pA on bilayer NaCl, Δz = -0.3
Å. The tip-height offset Δz for each panel is provided with respect to the STM setpoint, and positive (negative) values of Δz denote tip approach (retraction) from the STM setpoint. f and g show the same molecule at the same adsorption site, which is next to a trilayer NaCl island. The bright and dark features in the trilayer NaCl island in g correspond to Cl -and Na + ions, respectively.
Scale bars: 10 Å (f) and 5 Å (g). To experimentally explore the electronic structure of 5, we used bilayer NaCl films on coinage metal surfaces to electronically decouple the molecule from the metal surfaces. Before presenting the experimental findings, we summarize the results of our theoretical calculations performed on 5 in the neutral charge state (denoted as 5 0 ).
We start by performing DFT calculations on 5 0 in the gas phase. Geometry optimization performed at the spin-unrestricted UB3LYP/6-31G level of theory leads to one local minimum, 5OS, the geometry of which corresponds to the open-shell resonance structure of 5 (Fig. , the label OS denotes open-shell).
The triplet electronic configuration of 5OS is the lowest-energy state, with the openshell singlet configuration 90 meV higher in energy. Geometry optimization performed at the restricted closed-shell RB3LYP/6-31G level reveals two local minima, 5para and 5ortho, the geometries of which (Fig. ) exhibit bond length alternations in line with the presence of a para-or an ortho-QDM moiety, respectively, in the as-indacene core of the closed-shell resonance structures of 5 (Fig. ) .
Relative to 5OS in the triplet configuration, 5para and 5ortho are 0.40 and 0.43 eV higher in energy, respectively. Additional DFT results are shown in Supplementary Fig. . To gain more accurate insights into the theoretical electronic structure of 5, we performed multireference perturbation theory calculations (Supplementary Fig. ) based on quasi-degenerate second-order n-electron valence state perturbation theory (QD-NEVPT2).
In so far as the order of the ground and excited states are concerned, the results of QD-NEVPT2 calculations qualitatively match with DFT calculations. For 5OS, the triplet configuration remains the lowest-energy state, with the open-shell singlet configuration 60 meV higher in energy. The energy differences between the open-and closed-shell states are substantially reduced in QD-NEVPT2 calculations, with 5para and 5ortho only 0.11 and 0.21 eV higher in energy, respectively, compared to 5OS in the triplet configuration.
We also performed nucleus-independent chemical shift calculations to probe local aromaticity of 5 in the openand closed-shell states. While 5OS in the triplet configuration exhibits local aromaticity at the terminal benzenoid rings, 5OS in the open-shell singlet configuration, 5para and 5ortho all display antiaromaticity (Supplementary Fig. ).
The choice of the insulating surface determines the charge state of 5: while 5 adopts neutral charge state on the high work function bilayer NaCl/Au(111) surface (irrespective of its openor closed-shell state, Supplementary Fig. ), 5 exhibits charge bistability between 5 0 and the anionic state 5 -1 on the lower work function bilayer NaCl/Ag(111) and Cu(111) surfaces (Supplementary Figs. ).
In the main text, we focus on the characterization of 5 on bilayer NaCl/Au(111). Characterization of charge bistable 5 is reported in Supplementary Figs. . We first describe experiments on 5 on bilayer NaCl/Au(111), where 5 exhibits a geometry corresponding to the calculated 5OS geometry, and an open-shell electronic configuration.
We compare the experimental data on this species to calculations on 5OS with a triplet configuration, as theory predicts a triplet ground state for 5OS. For 5OS, the calculated frontier orbitals correspond to the SOMOs ψ1 and ψ2 (Fig. ), whose spin up levels are occupied and the spin down levels are empty.
Figure shows the DFT-calculated bond lengths of 5OS, where the two salient features, namely, the small difference in the bond lengths within each ring and the notably longer bond lengths in the pentagonal rings, agree with the open-shell resonance structure of 5 (Fig. ). Figure shows an AFM image of 5 adsorbed on bilayer NaCl/Au(111) that we assign as 5OS, where the bond-order differences qualitatively correspond to the calculated 5OS geometry (discussed and compared to the closed-shell state below).
Differential conductance spectra (dI/dV(V), where I and V denote the tunneling current and bias voltage, respectively) acquired on assigned 5OS exhibit two peaks centered at -1.5 V and 1.6 V (Fig. ), which we assign to the positive and negative ion resonances (PIR and NIR), respectively. Figure shows the corresponding STM images acquired at the onset (V = -1.2
V/1.3 V) and the peak (V = -1.5 V/1.6 V) of the ionic resonances. To draw a correspondence between the STM images and the molecular orbital densities, we consider tunneling events as many-body electronic transitions between different charge states of 5OS (Fig. ). Within this framework, the PIR corresponds to transitions between 5 0 and the cationic state 5 .
At the onset of the PIR at -1.2 V, an electron can only be detached from the SOMO ψ1 and the corresponding STM image at -1.2 V shows the orbital density of ψ1. Increasing the bias to the peak of the PIR at -1.5 V, it becomes possible to also empty the SOMO ψ2, such that the corresponding STM image shows the superposition of ψ1 and ψ2, that is, |ψ1| 2 + |ψ2| 2 (ref.
). Similarly, the NIR corresponds to transitions between 5 0 and 5 -1 . At the NIR onset of 1.3 V, only electron attachment to ψ2 is energetically possible. At 1.6 V, electron attachment to ψ1 also becomes possible, and the corresponding STM image shows the superposition of ψ1 and ψ2. The observation of the orbital densities of SOMOs, and not the hybridized HOMO and LUMO, proves the open-shell ground state of assigned 5OS.
Measurements of the monoradical species with a doublet ground state are shown in Supplementary Fig. . Unexpectedly, another species of 5 was also experimentally observed that exhibited a closedshell ground state. In contrast to 5OS, where the frontier orbitals correspond to the SOMOs ψ1 and ψ2, DFT calculations predict orbitals of different shapes and symmetries for 5para and 5ortho, denoted as α and β and shown in Fig. .
For 5ortho, α and β correspond to HOMO and LUMO, respectively. The orbitals are inverted in energy and occupation for 5para, where β is the HOMO and α is the LUMO. Fig. shows an AFM image of 5 that we assign as 5para. We experimentally infer its closed-shell state first by using qualitative bond order discrimination by AFM.
In high-resolution AFM imaging, chemical bonds with higher bond order are imaged brighter (that is, with higher frequency shift Δf) due to stronger repulsive forces, and they appear shorter . In Fig. , we label seven bonds whose bond orders show significant qualitative differences in the calculated 5ortho, 5para (Fig. ) and 5OS (Fig. ) geometries.
In 5para, the bonds b and d exhibit a higher bond order than a and c, respectively. This pattern is reversed for 5ortho, while the bond orders of the bonds a-d are all similar and small for 5OS. Furthermore, in 5para bond f exhibits a higher bond order than e, while in 5ortho and 5OS bonds e and f exhibit similar bond order (because they belong to Clar sextets).
Finally, the bond labeled g shows a higher bond order in 5para than in 5ortho and 5OS. The AFM image of assigned 5para shown in Fig. indicates higher bond orders of the bonds b, d and f compared to a, c and e, respectively. In addition, the bond g appears almost point-like and with enhanced Δf contrast compared to its neighboring bonds, indicative of a high bond order (see Supplementary Fig. for height-dependent measurements).
These observations concur with the calculated 5para geometry (Fig. ). Importantly, all these distinguishing bond-order differences are distinctly different in the AFM image of 5OS shown in Fig. , which is consistent with the calculated 5OS geometry (Fig. ). In the AFM images of 5OS (Fig. and Supplementary Fig. ), the bonds a-d at the pentagon apices appear with similar contrast and apparent bond length.
The bonds e and f at one of the terminal benzenoid rings also exhibit similar contrast and apparent bond length, while the central bond g appears longer compared to assigned 5para. Further compelling evidence for the closed-shell state of assigned 5para is obtained by STM and STS. dI/dV(V) spectra acquired on an assigned 5para species exhibit two peaks centered at -1.4 V (PIR) and 1.6 V (NIR) (Fig. ).
STM images acquired at these biases (Fig. ) show the orbital densities of β (-1.4 V) and α (1.6 V). First, the observation of α and β as the frontier orbitals of this species, and not the SOMOs, strongly indicates its closed-shell state. Second, consistent with AFM measurements that indicate good correspondence to the calculated 5para geometry, we observe β as the HOMO and α as the LUMO.
For 5ortho, α should be observed as the HOMO and β as the LUMO. We did not observe molecules with the signatures of 5ortho in our experiments. We observed molecules in open-(5OS, Fig. ) and closed-shell (5para, Fig. ) states in similar occurrence after their generation from 6 on the surface. We could also switch individual molecules between open-and closed-shell states as shown in Fig. and Supplementary Fig. .
To this end, a change in the adsorption site of a molecule was induced by STM imaging at ionic resonances, which often resulted in movement of the molecule. The example presented in Fig. shows a molecule that was switched from 5para to 5OS and back to 5para. The switching is not directed, that is, we cannot choose which of the two species will be formed when changing the adsorption site, and we observed 5OS and 5para in approximately equal yields upon changing the adsorption site.
The molecule in Fig. is adsorbed on top of a defect that stabilizes its adsorption geometry on bilayer NaCl. At defect-free adsorption sites on bilayer NaCl, that is, without a third layer NaCl island or atomic defects in the vicinity of the molecule, 5 could be stably imaged neither by AFM nor by STM at ionic resonances (Supplementary Fig. ).
Without changing the adsorption site, the state of 5 (open-or closedshell) never changed, including the experiments on bilayer NaCl/Ag(111) and Cu(111), on which the charge state of 5 could be switched (Supplementary Figs. ). Also on these lower work function surfaces, both open-and closed-shell species were observed for 5 0 and both showed charge bistability between 5 0 (5OS or 5para) and 5 -1 (Supplementary Figs. ).
The geometrical structure of 5 -1 probed by AFM, and its electronic structure probed by STM imaging at the NIR (corresponding to transitions between 5 -1 and the dianionic state 5 -2 ), are identical within the measurement accuracy for the charged species of both 5OS and 5para. When cycling the charge state of 5 between 5 0 and 5 -1 several times, we always observed the same state (5OS or 5para) when returning to 5 0 , provided the molecule did not move during the charging/discharging process.
Based on our experimental observations we conclude that indeno[1,2-a]fluorene (5), the last unknown indenofluorene isomer, can be stabilized in and switched between an open-shell (5OS) and a closed-shell (5para) state on NaCl. For the former, both DFT and QD-NEVPT2 calculations predict a triplet electronic configuration.
Therefore, 5 can be considered to exhibit the spin-crossover effect, involving magnetic switching between high-spin (5OS) and low-spin (5para) states, coupled with a reversible structural transformation. So far, the spin-crossover effect has mainly only been observed in transition-metal-based coordination compounds with a near-octahedral geometry .
The observation that the switching between open-and closedshell states is related to changes in the adsorption site but is not achieved by charge-state cycling alone, indicates that the NaCl surface and local defects facilitate different electronic configurations of 5 depending on the adsorption site.
Gas-phase QD-NEVPT2 calculations predict that 5OS is the ground state, and the closed-shell 5para and 5ortho states are 0.11 and 0.21 eV higher in energy. The experiments, showing bidirectional switching between 5OS and 5para, indicate that a change in the adsorption site can induce sufficient change in the geometry of 5 (leading to a corresponding change in the ground state electronic configuration) and thus induce switching.
Switching between open-and closed-shell states in 5 does not require the breaking or formation of covalent bonds , but a change of adsorption site on NaCl where the molecule is physisorbed. Our results should have implications for single-molecule devices, capitalizing on the altered electronic and chemical properties of a system in π-diradical open-shell and closed-shell states such as frontier orbital and singlet-triplet gaps, and chemical reactivity.
For possible future applications as a single-molecule switch, it might be possible to also switch between open-and closed-shell states by changing the local electric field, such as by using chargeable adsorbates . Scanning probe microscopy measurements and sample preparation. STM and AFM measurements were performed in a home-built system operating at base pressures below 1×10 -10 mbar and a base temperature of 5 K. Bias voltages are provided with respect to the sample.
All STM, AFM and spectroscopy measurements were performed with carbon monoxide (CO) functionalized tips. AFM measurements were performed in non-contact mode with a qPlus sensor . The sensor was operated in frequency modulation mode with a constant oscillation amplitude of 0.5 Å. STM measurements were performed in constantcurrent mode, AFM measurements were performed in constant-height mode with V = 0 V, and I(V) and Δf(V) spectra were acquired in constant-height mode.
Positive (negative) values of the tip-height offset Δz represent tip approach (retraction) from the STM setpoint. All dI/dV(V) spectra are obtained by numerical differentiation of the corresponding I(V) spectra. STM and AFM images, and spectroscopy curves, were post-processed using Gaussian low-pass filters.
Au(111), Ag(111) and Cu(111) surfaces were cleaned by iterative cycles of sputtering with Ne + ions and annealing up to 800 K. NaCl was thermally evaporated on Au(111), Ag(111) and Cu(111) surfaces held at 323 K, 303 K and 283 K, respectively. This protocol results in the growth of predominantly bilayer (100)-terminated islands, with a minority of trilayer islands.
Sub-monolayer coverage of 6 on surfaces was obtained by flashing an oxidized silicon wafer containing the precursor molecules in front of the cold sample in the microscope. CO molecules for tip functionalization were dosed from the gas phase on the cold sample. Density functional theory calculations. DFT was employed using the PSI4 program package .
All molecules with different charge (neutral and anionic) and electronic (open-and closed-shell) states were independently investigated in the gas phase. The B3LYP exchangecorrelation functional with 6-31G basis set was employed for structural relaxation and singlepoint energy calculations. The convergence criteria were set to 10 −4 eV Å −1 for the total forces and 10 −6 eV for the total energies.
Multireference calculations. Multireference calculations were performed on the DFToptimized geometries using the QD-NEVPT2 level of theory , with three singlet roots and one triplet root included in the state-averaged calculation. A (10,10) active space (that is, 10 electrons in 10 orbitals) was used along with the def2-TZVP basis set .
Increasing either the active space size or expanding the basis set resulted in changes of about 50 meV for relative energies of the singlet and triplet states. These calculations were performed using the ORCA program package . Nucleus-independent chemical shift (NICS) calculations. Isotropic nucleus-independent chemical shift values were evaluated at the centre of each ring using the B3LYP exchangecorrelation functional with def2-TZVP basis set using the Gaussian 16 software package .
Starting materials (reagent grade) were purchased from TCI and Sigma-Aldrich and used without further purification. Reactions were carried out in flame-dried glassware and under an inert atmosphere of purified Ar using Schlenk techniques. Thin-layer chromatography (TLC) was performed on Silica Gel 60 F-254 plates (Merck).
Column chromatography was performed on silica gel (40-60 µm). Nuclear magnetic resonance (NMR) spectra were recorded on a Bruker Varian Mercury 300 or Bruker Varian Inova 500 spectrometers. Mass spectrometry (MS) data were recorded in a Bruker Micro-TOF spectrometer. The synthesis of compound 6 was developed following the two-step synthetic route shown in Supplementary Fig. , which is based on the preparation of methylene-bridge polyarenes by means of Pd-catalyzed activation of benzylic C-H bonds .
Supplementary Figure | Synthetic route to obtain compound 6. The complex Pd2(dba)3 (20 mg, 0.02 mmol) was added over a deoxygenated mixture of 1,3-dibromo-2,4-dimethylbenzene (9, 100 mg, 0.38 mmol), boronic acid 10 (178 mg, 1.14 mmol), K2CO3 (314 mg, 2.28 mmol) and XPhos (35 mg, 0.08 mmol) in toluene (1:1, 10 mL), and the resulting mixture was heated at 90 °C for 2 h.
After cooling to room temperature, the solvents were evaporated under reduced pressure. The reaction crude was purified by column chromatography (SiO2; hexane:CH2Cl2 9:1) affording 11 (94 mg, 76%) as a colorless oil. The complex Pd(OAc)2 (7 mg, 0.03 mmol) was added over a deoxygenated mixture of terphenyl 11 (90 mg, 0.27 mmol), K2CO3 (114 mg, 0.83 mmol) and ligand L (26 mg, 0.06 mmol) in NMP (2 mL).
The resulting mixture was heated at 160 °C for 4 h. After cooling to room temperature, H2O (30 mL) was added, and the mixture was extracted with EtOAc (3x15 mL). The combined organic extracts were dried over anhydrous Na2SO4, filtered, and evaporated under reduced pressure. The reaction crude was purified by column chromatography (SiO2; hexane:CH2Cl2 9:1) affording compound 6 (8 mg, 11%) as a white solid. in AFM imaging due to their reduced adsorption height compared to the rest of the carbon atoms.
We attribute this observation to the significantly different lattice parameter of Cu(111) (2.57 Å) compared to Au(111) and Ag(111) (2.95 Å and 2.94 Å, respectively) , such that the apical carbon atoms of the pentagonal rings of 5 adsorb on the on-top atomic sites on Au(111) and Ag(111), but not on Cu(111).
Our speculation is based on a previous study of polymers of 1 on Au(111) by Di Giovannantonio et al. , where both tilted and planar individual units of 1 were observed depending on whether the apical carbon atoms of the pentagonal rings in 1 adsorbed on the on-top or hollow sites of the surface, respectively.
Given the strong molecule-metal interaction, we found no electronic state signatures of 5 on all three metal surfaces. STM set point for AFM images: V = 0. e, Frontier orbital spectrum of 5 -1 . In the anionic state, ψ2 becomes doubly occupied and ψ1 is the SOMO. Filled and empty circles denote occupied and empty orbitals, respectively.
For each panel, zero of the energy axis has been aligned to the respective highest-energy occupied orbital.

Answer: Open-shell π-diradical state and closed-shell state with a para-quinodimethane moiety.


Input: How does the receptive field size affect the completion of shapes?

Context:
Paper Info

Title: CONTOUR COMPLETION USING DEEP STRUCTURAL PRIORS
Publish Date: 9 Feb 2023
Author List: Ali Shiraee, Morteza Rezanejad, Mohammad Khodadad, Dirk Walther, Hamidreza Mahyar

Figure

Figure 1: Just by looking at subfigure (a), we, as humans, can easily perceive a shape like the one in subfigure (b).This is an extraordinary capability of our human brain and in this paper, we tried to see whether convolutional neural networks can show such capabilities.
Figure 2: The trajectory from random noise X N to the incomplete image X I in image space.The network will pass a completed version of the image, X C , throughout this trajectory.
Figure 4: This figure shows our iterative process to complete the fragmented contours of an image given as input to our pipeline.
Figure 5: This example shows how different scores change throughout a single run.All three scores change in the range of [0, 100].Our goal is to maximize reconstruction_score and minimize the overfit_score, but we should consider that the minimization lower bound is data dependent and is not zero.
Figure 6: Evolutionary process of the deep structure prior.The right column shows the incomplete shapes given to the model and the rest of the columns show how the model is overfitting gradually to produce the incomplete shapes.In each column, we are showing an intermediate iteration of this process.The loss-term setup enables our pipeline to let the completed image appears during this iterative process.
Average MSE and IoU values between the incomplete (Raw) images, the output of DIP and DSP methods, and ground truth for each image are provided in this table.
For this experiment, we ran the model over a subset of complex dataset with 500 incomplete images at various levels of alpha for 250 iterations.After the image completion is done, we compared the evaluation metrics between the completed image and the ground truth to examine the performance of the model for different values of alpha.
In this table, we show the effect of the receptive filter size on our algorithm's capability to fill in bigger gap sizes.The numbers in this table are showing the percentage of the time that DIP was successful to complete shapes with each gap size and corresponding receptive field size.As predicted, the bigger the filter size, the more successful the algorithm is in filling in the gaps.

abstract

Humans can easily perceive illusory contours and complete missing forms in fragmented shapes. This work investigates whether such capability can arise in convolutional neural networks (CNNs) using deep structural priors computed directly from images. In this work, we present a framework that completes disconnected contours and connects fragmented lines and curves.
In our framework, we propose a model that does not even need to know which regions of the contour are eliminated. We introduce an iterative process that completes an incomplete image and we propose novel measures that guide this to find regions it needs to complete. Our model trains on a single image and fills in the contours with no additional training data.
Our work builds a robust framework to achieve contour completion using deep structural priors and extensively investigate how such a model could be implemented.

Introduction

The human visual system is used to seeing incomplete outlines. Our brains can effortlessly group visual elements and fragmented contours that seem to be connected to each other. This power enables us to make shapes, organize disconnected visual features, and even properties of 3D surfaces when projected on 2D planes.
demonstrated how early vision may quickly complete partially-occluded objects using monocular signals. This capability of perceptual grouping has been studied in vision science for decades . Although there has been some work on perceptual grouping in the past couple of years, it has been less studied in the past decade due to the enormous progress of deep neural networks and their success in dealing with the pixel-by-pixel inference of images.
Different types of lines and curves have been studied to maximize the connectivity of two broken ends in the planer contour completion problem . Different types of lines and curves have been studied to maximize the connectivity of two broken ends in the planer contour completion problem. Geometry-based constraints can be utilized to address some challenges of contour completion problems, such as smoothness and curvature consistency .
However, such approaches only work for simple, smooth contours and usually fail in more complex settings. On the other hand, we currently have deep models that could easily take an incomplete image and complete the missing regions using enough training data . The amazing capability of such models especially those that are trained on different modalities with millions or billions of training data raises the question of whether we need such a large amount of training to perceive all the visual cues that are present in an image, which underlies visual perception by humans.
In human vision, Gestalt psychology suggests that our brain is designed to perceive structures and patterns that are grouped by some known rules. In this work, we show that some perceptual structures can also be learned from the image itself directly using architectures that enable such learning. Earlier work has shown  This is an extraordinary capability of our human brain and in this paper, we tried to see whether convolutional neural networks can show such capabilities.
that some forms of perceptual grouping can be achieved using computational models, such as stochastic completion fields . This type of learning resonates with some of the Gestalt perceptual grouping principles including "proximity", "good continuation" and "similarity". In scenarios where color and/or texture are present, the cue of "similarity" helps us group regions with consistent patterns .
When color and texture are present, they provide a collection of rich information for such cues. In the present article, we probe convolutional neural networks in a scenario where both are absent, and the neural network is dealing with just forms and shapes. Specifically, we explore whether the convolutional neural network architecture itself can give rise to some of these grouping cues when they are fed just contours and shapes alone.
For years, neural networks have been treated as black boxes that can generalize images very well to multiple classes when there are enough training exemplars. One of the reasons that neural networks are trained on many exemplars is to avoid the problem of overfitting. On the other hand, we know that CNNs that generalize well to large classes of exemplars can easily overfit when those class labels are randomly permuted .
Inspired by this observation, suggest that image priors can be learned to a large extent through a generator network architecture that is solely trained on a single image. This encouraged us to take a deeper look at what structural information can be learned from a single-shape image and whether we can reconstruct some of those perceptual grouping capabilities using a generator network.
Inspired by , in this work, we adopt a novel training regime to complete shapes and contours where we use a UNet architecture with random initial weights and try to complete the contours within a single image without any training data. In our model, we imagine that the input image (i.e., the only image used to update the model's weights) is an image of fragmented contours.
In this work, instead of training the model on multiple images fetched from a big image dataset, we imagine a random fixed tensor noise image as input to this model. At each iteration, the random noise tensor is inferred through our generative network and the network produces an outcome image. We introduce a novel loss function that enables this network to complete contours.
This process repeats, and the weights of our network are updated gradually based on this loss function, which is an energy term defined based on the input image and the output of the network. The model will reconstruct the missing structures i.e., group fragmented contours that perceptually seem to be connected, before it fully overfits to the incomplete input image.
Contributions of our work are summarized as follows: 1. In our pipeline, we propose a novel algorithm that enables us to complete contours that appear to be connected to each other in an illusory form. 2. Our model is trained on just one single query image and does not need any training data. 3. Our model does not need to know which regions of the image are masked or occluded, i.e., we remove the dependency of the algorithm on the guiding mask (a guiding mask is a mask that informs the model on where the missing regions are located at).
We also introduce two metrics to produce a stopping criterion to know when to stop training before the model fully overfits to the incomplete image, i.e., we guide the model to stop when the completed image is produced.

Methods

Our eyes are trained to predict a missing region of an occluded object within a scene. We can easily perceive or make guesses about parts of objects or shapes that we do not necessarily see. Even when we are looking at an image, we might guess about the shape, property, or other attributes of an unknown object within a scene.
Such capability extends beyond just known objects or shapes. We can look at a disconnected set of contours and guess what the connected form may look like. This capability is rooted in our prior knowledge about the world. (see Figure ). In this work, we aim to achieve a similar capability using deep generative networks.
Most neural networks that we work with these days are trained with a massive amount of data and one might think that this is the only way that a neural network can obtain prior information. Authors of Deep Image Prior (DIP) suggest that the convolutional architecture can capture a fair amount of information about image distribution.
They show that the hourglass architectures like UNet can show some good performances in some inverse problems such as image denoising, super-resolution, and inpainting. In this work, we focus on completing fragmented contours end-to-end just by using a single image. To be able to address this problem, we first look at a similar problem in image editing, known as image inpainting.
Image inpainting is the task of completing an image where some regions of that image are covered or filtered by a mask. In image inpainting, the generative model receives a masked image with the mask that guides the algorithm to fill in those missing regions. Although in the problem of contour completion, we have a very similar goal, the additional challenge that we suffer from is that we do not necessarily have a mask that covers the regions of interest for us.
For example, when we look at Figure (left), we are not provided that which regions of the image are incomplete by a guiding mask. Our brain figures this out just by looking at the form and predicting those missing regions. Inspired by the image inpainting work of DIP , we propose a novel algorithm for the contour completion problem (see Figure ), where, unlike DIP, we do not have a guiding mask to know where to fill in the missing regions of our disconnected contours.
Let us assume that we are given a degraded image x I containing a fragmented contour. We propose an iterative process (see Figure ) that can connect those discontinuities and glue those fragmented pieces together as follows. We propose an hour-glass model structure (f ) that is initially set up with completely random parameters (θ 0 ) at first.
Through an iterative process, we start feeding our network with a fixed random tensor noise z signal and obtain the inferred output (f (z)) from that network. We then back-propagate the difference between the inferred output and the incomplete image to the network. We then repeat this process until the difference between the generated outcome of the network (f θ (z)) and the incomplete image (x I ) gets smaller and smaller and finally overfits the incomplete image (x I ).
In this work, we propose a novel error metric to backpropagate in the model and update its weights. we set the metric in a way that enables us to complete the incomplete image before it overfits the incomplete image. This is where the magic of our algorithm happens. We also propose a stopping criterion, so that when the image is complete, we no longer overfit the outcome of the model and instead produce a plausible connected set of fragmented contour pieces.
As illustrated in Figure , this trajectory will pass through a complete version of the image in image space, which is close to the actual connected ground truth x gt , which we do not have access to directly.

Energy Function

We can model this iterative process mathematically by maximizing a posterior distribution. Let us assume that the optimal image x * that we want to achieve is on a path that connects a random tensor noise z to the incomplete image x I . With this assumption, we can eventually overfit any random tensor noise to the incomplete image x I , and we can formulate the posterior distribution of our wanted optimal image x * as follows:
No Prior To better recapitulate what we want to achieve using our generative model, we solve an energy minimization problem on the parameter space of the model, rather than explicitly working with probability distributions and optimizing on x (image space). Thus, we solve an energy minimization problem that incorporates the incomplete image (x I ) and model parameters (f θ (z)):
As shown in Figure , the pipeline starts from a random initialized set of parameters θ and updates those weights until it reaches a local minimum θ * . The only information provided for the network is the incomplete image x I . When we reach the optimal θ * , the completed image is obtained as x * = f θ * (z) where z is random tensor noise.
In this work, we use a U-Net architecture with skip connections as the generator model. " As we mentioned previously, in this work we were inspired by an earlier work known as Deep Image Prior (DIP) . In this work, the authors suggested a mean-squared-error loss term that enables the network to compare the output of the generator to the incomplete input image:
where x I is the incomplete image with missing pixels in correspondence of a binary mask m ∈ {0, 1} H×W and operator is for point-wise multiplication of two image matrices. In the inpainting tasks, the existence of a mask is essential as the algorithm needs to know where to fill in the missing area, whereas, in our work, we wanted to know whether the network can perform completion on its own without the need for the mask.
In other words, is it possible for the network to predict where to fill in at the same time that it is trying to reconstruct the incomplete image through the iterative process? To answer this question, we tried to solve a much harder problem in which the mask is not provided to the model and the model is agnostic to it.
To better understand how a solution could be hypothesized for this problem, we first imagine that we want to consider all the available regions in our image that could be potential places to fill in, i.e., we set the mask in the previous formula 1 to be equal to the incomplete image x I . This is problematic as the model quickly tries to fill in all white space and quickly reconstructs the incomplete image by doing so.
On the other hand, we can take the inverse problem of the current problem, where the model tries to just fill in the regions that fragmented contour lives in. Taking these two at the same time, we came up with a novel loss term for energy minimization term that helps us remove the need for the mask in the case of the contour completion problem:
In this term, we introduce a linear combination of the two loss terms, where one focuses on reconstructing the missing regions in the foreground, and one focuses on avoiding inpainting regions in the background. The logic behind this is that, if we assume the original image to be representative of the mask, then the model tries to reconstruct in all white regions (the foreground), and in the inverse problem we just want to reconstruct the regions that are already part of the ground truth.

Stopping Criteria

As shown in Figure , knowing when to stop iterating to the over-fitted model is a key to obtaining a completed shape. Therefore, we equipped our model with a criterion that uses two individual novel terms to know when to stop and output the result of the network. These two metrics expand the capability of the generator network beyond what it does currently and achieve a full end-to-end contour completion model that trains and infers on a single image of divided contour fragments.
These new terms are: reconstruction_score (ρ) and overfit_score (ω).

Reconstruction Score

The first score that this paper suggests is the reconstruction score, i. e., we have to make sure that the model is trained enough that it can reconstruct at least the entire set of fragmented contours within the image. This is a trivial score and to compute the reconstruction_score (ρ), we apply a k-dimensional tree (KDTree) nearest-neighbor lookup to find the ratio of points in the original incomplete image (x 0 ).
This score ranges from [0 − 100].

Overfit Score

It is evident that the model overfits the fragmented contours. This is due to the fact that the error in our loss term is minimized as the x overfits to x I , i. e., replacing x with x I in the loss term would give us zero. As we hypothesize iterative process also produces the complete image before it overfits to the incomplete image, we can imagine that at some point the image is complete (x C ) and does not need to be fine-tuned any more to overfit to x I .
We suggest a new score called overfit_score. overfit_score determines how much of the reconstructed outcome is over the number of pixels that are already in the incomplete image (x I ). To compute the overfit_score (ω), we apply a k-dimensional tree (KDTree) nearest-neighbor lookup of points in the outcome of the input image and see what portions of those points are novel and not already in the incomplete image (x I ).
Similar to reconstruction_score, the overfit_score also ranges from [0 − 100]. Our goal is to maximize reconstruction_score and minimize the overfit_score, but we should consider that the minimization lower bound is data dependent and is not zero.

Combined Score

To be able to find the best possible set of completed contours, we combine the two and have a loop that tries to achieve close to full reconstruction and avoids over-fitting at the same time. This is what we call an "ideal" stopping point in the contour completion problem. In each run throughout all iterations, we pick an output of the network that minimizes a dissimilarity term:
where δ represents our dissimilarity score. The reconstruction_score and overfit_score are obtainable given network output and the incomplete image. Ideally, we want to achieve an output image that has a reconstruction_score equal to 100 and an overfit_score of γ which is a hyperparameter that is dependent on the image and complexity of the shape.
Empirically, we observed that this value is highly correlated with the distance between gaps that are observed in fragmented contours, i. e., the larger the gap results in a larger γ value. We will discuss this in more detail in the next section (see Section 3). For one sample image, we computed the two metrics reconstruction_score and overfit_score and the combined value of dissimilarity (δ) and showed how these values change (see Figure ).
Our initial observations show that the reconstruction_score will increase to 100 quickly for the incomplete image indicating that the already existing fragments of the contours have been reconstructed in the output. However, as mentioned previously, we cannot solely rely on this score since we also want to minimize the overfitting.
Remember that our goal is to produce an output that: a) preserves the original contours in the incomplete image and b) fills in the gaps between the fragmented contours. It is evident that overfit_score decreases throughout an iterative run of our process until it reaches zero. The dissimilarity will also decrease along with the overfit to a point, then it will increase, as the model tries to reproduce the incomplete image.
This is where an ideal γ value can be picked, i.e., where to stop when the reconstruction is good but we have not done a full overfit to the incomplete image. Thus, one should pick the value of γ empirically in the scenario that the ground truth is not available, whereas, assuming that the ground truth is available, we can easily compute the best γ value.
In our experiments, we tried two datasets of images with different gap sizes. We observed that the best the γ for one set of samples is ∼ 5 (the set with shorter gaps) while it is ∼ 23 for samples from the other set, i. e, the set with longer gaps (see Figure for some completed examples).

Experiments and Results

Performing unsupervised contour completion is a difficult task to benchmark as one can never know what fragments exactly are connected to each other in a real-world scenario. This makes the problem of contour completion a hard problem to solve. In this paper, we tried to create artificial shapes that are occluded by some masks and then tried to see if our model can regenerate the missing pieces and glue those divided contours together.
To demonstrate our model's behavior, we will conduct experiments on datasets created for this task and will report on them in this section. To compare network results in different settings, we will use pixel-wise Mean Squared Error (MSE) and Intersection over Union (IoU) between the produced result of the network and unmasked ground truth data and the reconstructed image on black pixels (where contours live).

Data

We prepared two datasets, one labeled "Simple" and one "Complex", in accordance with the number of gaps in each shape. Both datasets contain nine different categories of shapes. In order to generate the Complex dataset, we used FlatShapeNet which is a dataset for the educational game Ariga. The dataset includes the following categories: Circle, Kite, Parallelogram, Rectangle, Rhombus, Square, Trapezoid, Triangle and Overlap.
The "overlap" category contains images that are made as a mixture of two shapes that are overlapping from the previous categories. These are some standard shapes with a few gaps in simple dataset, while the complex dataset has some hand-drawn shapes with fragmented lines and more gaps that produce more variety in general.
For each instance, a ground truth image is available for comparison. Most of our experiments have been conducted using the complex dataset in order to evaluate the generalization of our approach. For the analysis of how γ values should be set for each shape, we used the simple dataset as a reference.

Evaluation

In this section, we compare our model to the original Deep Image Prior (DIP) inpainting model. DIP's inpainting module accepts a degraded image and a binary mask corresponding to that image. In order to make a fair comparison, instead of providing a binary mask, we used the incomplete images both as input and as a mask in order to see whether it can produce a result similar to ours.
For DIP, we run the iterative process for a maximum number of 2500 iterations with the U-net backbone. We used the exact same architecture and setting in our model for a fair comparison. Using our ground truth dataset images, we calculate the MSE loss between the network output and ground truth during each iteration instead of relying on our stopping mechanism described in the previous section.
We then store the output with minimal loss throughout all the iterations. Finally, we select the best output among all iterations, report the MSE and IoU with the ground truth, and save the iteration number which resulted in the lowest MSE. Table compares the results that are obtained using the DIP method, the DSP method (ours), and the difference between raw images and the ground truth.
We have presented the average MSE-loss, average IoU, and the average number of iterations for the best output for different methods. As can be seen from the table, our model improves both MSE and IoU between the incomplete image and ground truth in fewer iterations. The DIP method can neither generate a better result than the raw image nor provide stopping criteria to prevent overfitting.
We provide a more detailed analysis of this result in Figure . As results show, our algorithm not only provides a much faster convergence but also consistently provides a better-completed image (consistently less MSE loss and better IoU), whereas it is challenging for the DIP method to accomplish better results without a guiding mask.
We compare MSE loss between the degraded raw images that these algorithms started with (shown in blue) (a) Mean Squared Error Loss: we clearly see that for almost all images, DSP (green) achieves a lower MSE than the incomplete images (blue) whereas, the DIP completed images either do not improve the MSE or even worsen that for the incomplete images.
Note that, the MSE is computed to an available ground truth image hidden from our methods (the lower is better). (b) Intersection Over Union: here, we are looking at the IoU metric that specifies the amount of intersection over the union between the obtained images and the ground truth data. Again, we see that DSP produces images that are much closer to the ground truth (in most cases) whereas the DIP can not achieve a similar result.
While we see few DIP completed images produce a better IoU than the degraded images (in terms of IoU), most of them are worse than the starting image (the higher is better). (c) The number of iterations that are needed for each algorithm to obtain its best results. Here, we see that DSP can quickly produce the best outcome with the least MSE loss whereas the DIP algorithm's best results are when we run the iterative process for more iterations (the lower is better).

Correlation of γ with the Gap Size

To better understand the γ parameter of our combined score, we conducted the following experiment. A total of 12000 samples were obtained by merging all of our images from the two datasets, simple and complex. As we have access to the ground truth for each degraded image in the combined dataset, we can easily calculate reconstruction_score and overfit_score for each degraded-ground truth pair.
As expected, we obtain a reconstruction_score of 100 for all samples, but the overfit_score varies among them. Intuitively, we hypothesized that an optimal value of overfit score should be intertwined with the total area of gaps. To test this hypothesis, we did the following experiment. We first define a function φ(x) which takes a binary, black and white image x and returns the number of black pixels in it.
Then we define a gap term as follows: where x I is the incomplete image and x gt is the ground truth. In this case, gap indicates the total area of the gap with respect to the whole shape. We found out that this term and the best result have a correlation of 97.43%. This indicates that the value of γ is highly correlated with the gap size, that is something expected in a way.

Effect of α

We conducted additional experiments concerning how α affects the quality of reconstruction. In the previous section, we defined Equation 2 as the loss term that guides our iterative process. The term α specifies the amount of emphasis the model should place on reconstructing missing regions, rather than filling in fragmented contours.
A lower α indicates a better grouping quality, as shown in Equation . However, we will not achieve completion if we remove the first term completely from the loss by setting α = 0. Therefore, the first term should be kept, but its weight should be very low in order to achieve a good completion. On the other hand, if we set α = 1 and omit the second term, we lose the contour completion regularization term and obtain the same output as a vanilla deep image prior, which does not complete shapes.

Effect of Receptive Field Size

To better understand the effect of receptive field size on our algorithm, we test the following hypothesis: can models with bigger receptive field size complete shapes with bigger gaps? In Table , we report showing the results of this experiment. As we can see, the bigger the receptive field size, the more complete shapes we can reconstruct using DSP.

Implementation Details

As shown in Figure , we use a model with 5 layers and 128 channels for downsampling and upsampling convolutions, and 64 channels for skip convolutions. The upsampling and downsampling modules use 3 × 3 filters, while the skip module uses 1 × 1 filters. In the upsample part of the network, the nearest neighbor algorithm is used.
We used 256 × 256 images with three channels in all of our experiments. In training, we use the MSE loss between the degraded image and the output of the network, and we optimize the loss using the ADAM optimizer and a learning rate equal to 0.01 . In our experiments, we also used α = 0.15 as an optimal proportion coefficient for reconstruction loss.

Conclusion

In this work, we introduced a novel framework for contour completion using deep structure priors (DSP). This work offers a novel notion of a maskless grouping of fragmented contours. In our proposed framework, we introduced a novel loss metric that does not require a strict definition of the mask. Instead, it lets the model learn the perceivable illusory contours and connects those fragmented pieces using a generator network that is solely trained on just the single incomplete input image.
Our model does not require any pre-training which demonstrates that the convolutional architecture of the hour-glass model is able to connect disconnected contours. We present an extended set of experiments that show the capability of our algorithm. We investigate the effect of each parameter introduced in our algorithm separately and show how one could possibly achieve the best result for their problem using this model.
In future work, we plan to extend this model and try to see how it performs with real images. In particular, we want to determine whether we can inpaint real-world photographs while retaining perceptually aware scene structures. The importance of shape in perception by deep neural networks has been highlighted in many adversarial examples to appearance-based networks .
The outcome of this work has strong potential to impact the designing and implementation of models that are robust to such perturbations.

Answer: Bigger receptive field size leads to more successful shape completion.