prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: What are the two ground states observed for indeno[1,2-a]fluorene on NaCl surfaces?

Read the following text and answer briefly.

Context 1: Paper Info

Title: Bistability between π-diradical open-shell and closed-shell states in indeno[1,2-a]fluorene
Publish Date: Unkown
Author List: Shantanu Mishra (from IBM Research Europe -Zurich), Manuel Vilas-Varela (from Department of Organic Chemistry, Center for Research in Biological Chemistry and Molecular Materials (CiQUS), University of Santiago de Compostela), Leonard-Alexander Lieske (from IBM Research Europe -Zurich), Ricardo Ortiz (from Donostia International Physics Center (DIPC)), Igor Rončević (from Department of Chemistry, University of Oxford), Florian Albrecht (from IBM Research Europe -Zurich), Diego Peña (from Department of Organic Chemistry, Center for Research in Biological Chemistry and Molecular Materials (CiQUS), University of Santiago de Compostela), Leo Gross (from IBM Research Europe -Zurich)

Figure

Fig  1 | Non-benzenoid non-alternant polycyclic conjugated hydrocarbons a, Classical nonbenzenoid non-alternant polycyclic conjugated hydrocarbons: pentalene, azulene and heptalene b, Generation of indacenes and indenoindenes through benzinterposition and benzannelation of pentalene, respectively Gray filled rings represent Clar sextets c, Closed-shell Kekulé (left) and openshell non-Kekulé (right) resonance structures of QDMs Note that meta-QDM is a non-Kekulé molecule All indenofluorene isomers, being derived through benzannelation of indacenes, contain a central QDM moiety d, Closed-shell Kekulé (top) and open-shell non-Kekulé (bottom) resonance structures of indenofluorenes Compared to their closed-shell structures, 1 and 5 gain two Clar sextets in the openshell structure, while 2-4 gain only one Clar sextet in the open-shell structure Colored bonds in d highlight the ortho-and para-QDM moieties in the two closed-shell Kekulé structures of 5  e, Scheme of on-surface generation of 5 by voltage pulse-induced dehydrogenation of 6 (C20H14) Structures 7 and 8 represent the two monoradical species (C20H13) 
Fig  2 | Characterization of open-shell indeno[1,2-a]fluorene on bilayer NaCl/Au(111) a, DFTcalculated wave functions of the frontier orbitals of 5OS in the triplet configuration for the spin up (occupied) level (isovalue: 0 002 e -Å -3 ) Blue and red colors represent opposite phases of the wave function b, Corresponding DFT-calculated spin density of 5OS (isovalue: 0 01 e -Å -3) Blue and orange colors represent spin up and spin down densities, respectively c, Probability density of the SOMOs of 5OS (isovalue: 0 001 e -Å -3 ) d, DFT-calculated bond lengths of 5OS e, Constant-height I(V) spectra acquired on a species of 5 assigned as 5OS, along with the corresponding dI/dV(V) spectra Open feedback parameters: V = -2 V, I = 0 17 pA (negative bias side) and V = 2 V, I = 0 17 pA (positive bias side) Acquisition position of the spectra is shown in Supplementary Fig 7 f, Scheme of many-body transitions associated to the measured ionic resonances of 5OS Also shown are STM images of assigned 5OS at biases where the corresponding transitions become accessible Scanning parameters: I = 0 3 pA (V = -1 2V and -1 5 V) and 0 2 pA (V = 1 3 V and 1 6 V)  g, Laplace-filtered AFM image of assigned 5OS STM set point: V = 0 2 V, I = 0 5 pA on bilayer NaCl, Δz = -0 3Å The tip-height offset Δz for each panel is provided with respect to the STM setpoint, and positive (negative) values of Δz denote tip approach (retraction) from the STM setpoint f and g show the same molecule at the same adsorption site, which is next to a trilayer NaCl island The bright and dark features in the trilayer NaCl island in g correspond to Cl -and Na + ions, respectively Scale bars: 10 Å (f) and 5 Å (g) 
Fig  3 | Characterization of closed-shell indeno[1,2-a]fluorene on bilayer NaCl/Au(111) a, DFTcalculated wave functions of the frontier orbitals of closed-shell 5 0 (isovalue: 0 002 e -Å -3 ) The wave functions shown here are calculated for the 5para geometry b, DFT-calculated bond lengths of 5ortho (top) and 5para (bottom) c, Constant-height I(V) spectra acquired on a species of 5 assigned as 5para, along with the corresponding dI/dV(V) spectra Open feedback parameters: V = -2 V, I = 0 15 pA (negative bias side) and V = 2 2 V, I = 0 15 pA (positive bias side) Acquisition position of the spectra is shown in Supplementary Fig  7  d, Scheme of many-body transitions associated to the measured ionic resonances of 5para Also shown are STM images of assigned 5para at biases where the corresponding transitions become accessible Scanning parameters: I = 0 15 pA (V = -1 5 V) and 0 2 pA (V = 1 7 V)  e, Laplace-filtered AFM image of assigned 5para STM set point: V = 0 2 V, I = 0 5 pA on bilayer NaCl, Δz = -0 7 Å  f, Selected bonds labeled for highlighting bond order differences between 5para and 5ortho For the bond pairs a/b, c/d and e/f, the bonds labeled in bold exhibit a higher bond order than their neighboring labeled bonds in 5para g, Laplace-filtered AFM images of 5 on bilayer NaCl/Cu(111) showing switching between 5OS and 5para as the molecule changes its adsorption position The faint protrusion adjacent to 5 is a defect that stabilizes the adsorption of 5  STM set point: V = 0 2 V, I = 0 5 pA on bilayer NaCl, Δz = -0 3Å  STM and STS data in c and d are acquired on the same species, while the AFM data in e is acquired on a different species Scale bars: 10 Å (d) and 5 Å (e,g) 
NMR (300 MHz, CDCl3) δ: 7 51 (m, 2H), 7 40 -7 28 (m, 5H), 7 27 -7 20 (m, 2H), 7 13 (d, J = 7 7 Hz, 1H), 2 07 (s, 3H), 1 77 (s, 3H) ppm  13C NMR-DEPT (75 MHz, CDCl3, 1:1 mixture of atropisomers) δ: 141 2 (C), 141 1 (C), 140 0 (C), 139 4 (2C), 137 5 (C), 137 4 (C), 136 0 (3C), 134 8 (C), 134 5 (C), 134 1 (C), 134 0 (C), 133 7 (C), 133 6 (C), 131 6 (CH), 131 2 (CH), 131 1 (CH), 130 7 (CH), 129 8 (CH), 129 7 (CH), 129 5 (CH), 129 4 (CH), 129 0 (CH), 128 9 (CH), 128 7 (2CH), 128 6 (2CH), 127 2 (CH), 127 1 (CH), 127 0 (CH), 126 9 (CH), 126 7 (CH), 126 6 (CH), 20 6 (CH3), 20 5 (CH3), 17 7 (CH3), 17 5 (CH3) ppm MS (APCI) m/z (%): 327 (M+1, 100) HRMS: C20H16Cl2; calculated: 327 0702, found: 327 0709 
NMR (500 MHz, CDCl3) δ: 7 93 (d, J = 7 6 Hz, 1H), 7 85 (d, J = 7 5 Hz, 1H), 7 78 (d, J = 7 7 Hz, 1H), 7 65 (d, J = 7 4 Hz, 1H), 7 61 (d, J = 7 5 Hz, 1H), 7 59 (d, J = 7 7 Hz, 1H), 7 47 (ddd, J = 8 4, 7 2, 1 1 Hz, 1H), 7 42 (dd, J = 8 1, 7 0 Hz, 1H), 7 35 (m, 2H), 4 22 (s, 3H), 4 02 (s, 3H) ppm  13C NMR-DEPT (125 MHz, CDCl3) δ: 144 1 (C), 143 3 (C), 142 3 (C), 141 9 (C), 141 8 (C), 141 2 (C), 138 2 (C), 136 5 (C), 127 0 (CH), 126 9 (CH), 126 7 (CH), 126 6 (CH), 125 3 (CH), 125 2 (CH), 123 6 (CH), 122 2 (CH), 119 9 (CH), 118 4 (CH), 37 4 (CH2), 36 3 (CH2) ppm MS (APCI) m/z (%): 254 (M+, 88) HRMS: C20H14; calculated: 254 1090, found: 254 1090 

abstract

Indenofluorenes are non-benzenoid conjugated hydrocarbons that have received great interest owing to their unusual electronic structure and potential applications in nonlinear optics and photovoltaics  Here, we report the generation of unsubstituted indeno[1,2-a]fluorene, the final and yet unreported parent indenofluorene regioisomer, on various surfaces by cleavage of two C-H bonds in 7,12-dihydro indeno[1,2-a]fluorene through voltage pulses applied by the tip of a combined scanning tunneling microscope and atomic force microscope 
On bilayer NaCl on Au(111), indeno[1,2a]fluorene is in the neutral charge state, while it exhibits charge bistability between neutral and anionic states on the lower work function surfaces of bilayer NaCl on Ag(111) and Cu(111)  In the neutral state, indeno[1,2-a]fluorene exhibits either of two ground states: an open-shell π-diradical state, predicted to be a triplet by density functional and multireference many-body perturbation theory calculations, or a closedshell state with a para-quinodimethane moiety in the as-indacene core 
Switching between open-and closed-shell states of a single molecule is observed by changing its adsorption site on NaCl  The inclusion of non-benzenoid carbocyclic rings is a viable route to tune the physicochemical properties of polycyclic conjugated hydrocarbons (PCHs)   Non-benzenoid polycycles may lead to local changes in strain, conjugation, aromaticity, and, relevant to the context of the present work, induce an open-shell ground state of the corresponding PCHs  
Many nonbenzenoid PCHs are also non-alternant, where the presence of odd-membered polycycles breaks the bipartite symmetry of the molecular network   Figure shows classical examples of non-benzenoid non-alternant PCHs, namely, pentalene, azulene and heptalene  Whereas azulene is a stable PCH exhibiting Hückel aromaticity ([4n+2] π-electrons, n = 2), pentalene and heptalene are unstable Hückel antiaromatic compounds with [4n] π-electrons, n = 2 (pentalene) and n = 3 (heptalene) 
Benzinterposition of pentalene generates indacenes, consisting of two isomers s-indacene and as-indacene (Fig  )  Apart from being antiaromatic, indacenes also contain proaromatic quinodimethane (QDM) moieties (Fig  ) , which endows them with potential open-shell character  While the parent s-indacene and asindacene have never been isolated, stable derivatives of s-indacene bearing bulky substituents have been synthesized  
A feasible strategy to isolate congeners of otherwise unstable non-benzenoid non-alternant PCHs is through fusion of benzenoid rings at the ends of the π-system, that is, benzannelation  For example, while the parent pentalene is unstable, the benzannelated congener indeno[2,1-a]indene is stable under ambient conditions (Fig  )  
However, the position of benzannelation is crucial for stability: although indeno[2,1a]indene is stable, its regioisomer indeno[1,2-a]indene (Fig  ) oxidizes under ambient conditions   Similarly, benzannelation of indacenes gives rise to the family of PCHs known as indenofluorenes (Fig  ), which constitute the topic of the present work 
Depending on the benzannelation position and the indacene core, five regioisomers can be constructed, namely, indeno [ Practical interest in indenofluorenes stems from their low frontier orbital gap and excellent electrochemical characteristics that render them as useful components in organic electronic devices  
The potential open-shell character of indenofluorenes has led to several theoretical studies on their use as non-linear optical materials and as candidates for singlet fission in organic photovoltaics   Recent theoretical work has also shown that indenofluorene-based ladder polymers may exhibit fractionalized excitations 
Fundamentally, indenofluorenes represent model systems to study the interplay between aromaticity and magnetism at the molecular scale   Motivated by many of these prospects, the last decade has witnessed intensive synthetic efforts toward the realization of indenofluorenes  Derivatives of 1-4 have been realized in solution , while 1-3 have also been synthesized on surfaces and characterized using scanning tunneling microscopy (STM) and atomic force microscopy (AFM), which provide information on molecular orbital densities , molecular structure and oxidation state  
With regards to the open-shell character of indenofluorenes, 2-4 are theoretically and experimentally interpreted to be closed-shell, while calculations indicate that 1 and 5 should exhibit open-shell ground states   Bulk characterization of mesitylsubstituted 1, including X-ray crystallography, temperature-dependent NMR, and electron spin resonance spectroscopy, provided indications of its open-shell ground state  
Electronic characterization of 1 on Au(111) surface using scanning tunneling spectroscopy (STS) revealed a low electronic gap of 0 4 eV (ref  )  However, no experimental proof of an openshell ground state of 1 on Au(111), such as detection of singly occupied molecular orbitals (SOMOs) or spin excitations and correlations due to unpaired electrons , was shown 
In this work, we report the generation and characterization of unsubstituted 5  Our research is motivated by theoretical calculations that indicate 5 to exhibit the largest diradical character among all indenofluorene isomers   The same calculations also predict that 5 should possess a triplet ground state 
Therefore, 5 would qualify as a Kekulé triplet, of which only a handful of examples exist   However, definitive synthesis of 5 has never been reported so far  Previously, Dressler et al  reported transient isolation of mesityl-substituted 5, where it decomposed both in the solution and in solid state , and only the structural proof of the corresponding dianion was obtained 
On-surface generation of a derivative of 5, starting from truxene as a precursor, was recently reported   STM data on this compound, containing the indeno[1,2-a]fluorene moiety as part of a larger PCH, was interpreted to indicate its open-shell ground state  However, the results did not imply the ground state of unsubstituted 5  Here, we show that on insulating surfaces 5 can exhibit either of two ground states: an open-shell or a closed-shell 
We infer the existence of these two ground states based on high-resolution AFM imaging with bond-order discrimination and STM imaging of molecular orbital densities   AFM imaging reveals molecules with two different geometries  Characteristic bond-order differences in the two geometries concur with the geometry of either an open-or a closed-shell state 
Concurrently, STM images at ionic resonances show molecular orbital densities corresponding to SOMOs for the open-shell geometry, but orbital densities of the highest occupied molecular orbital (HOMO) and lowest unoccupied molecular orbital (LUMO) for the closed-shell geometry  Our experimental results are in good agreement with density functional theory (DFT) and multireference perturbation theory calculations 
Finally, we observe switching between open-and closed-shell states of a single molecule by changing its adsorption site on the surface  Synthetic strategy toward indeno[1,2-a]fluorene  The generation of 5 relies on the solution-phase synthesis of the precursor 7,12-dihydro indeno[1,2-a]fluorene (6)  Details on synthesis and characterization of 6 are reported in Supplementary Figs 
  Single molecules of 6 are deposited on coinage metal (Au(111), Ag(111) and Cu(111)) or insulator surfaces  In our work, insulating surfaces correspond to two monolayer-thick (denoted as bilayer) NaCl on coinage metal surfaces  Voltage pulses ranging between 4-6 V are applied by the tip of a combined STM/AFM system, which result in cleavage of one C-H bond at each of the pentagonal apices of 6, thereby leading to the generation of 5 (Fig  ) 
In the main text, we focus on the generation and characterization of 5 on insulating surfaces  Generation and characterization of 5 on coinage metal surfaces is shown in Supplementary Fig    )  Blue and orange colors represent spin up and spin down densities, respectively  c, Probability density of the SOMOs of 5OS (isovalue: 0 001 e -Å -3 ) 
d, DFT-calculated bond lengths of 5OS  e, Constant-height I(V) spectra acquired on a species of 5 assigned as 5OS, along with the corresponding dI/dV(V) spectra  Open feedback parameters: V = -2 V, I = 0 17 pA (negative bias side) and V = 2 V, I = 0 17 pA (positive bias side)  Acquisition position of the spectra is shown in Supplementary Fig    f, Scheme of many-body transitions associated to the measured ionic resonances of 5OS 
Also shown are STM images of assigned 5OS at biases where the corresponding transitions become accessible  Scanning parameters: I = 0 3 pA (V = -1 2 V and -1 5 V) and 0 2 pA (V = 1 3 V and 1 6 V)  g, Laplace-filtered AFM image of assigned 5OS  STM set point: V = 0 2 V, I = 0 5 pA on bilayer NaCl, Δz = -0 3
Å  The tip-height offset Δz for each panel is provided with respect to the STM setpoint, and positive (negative) values of Δz denote tip approach (retraction) from the STM setpoint  f and g show the same molecule at the same adsorption site, which is next to a trilayer NaCl island  The bright and dark features in the trilayer NaCl island in g correspond to Cl -and Na + ions, respectively 
Scale bars: 10 Å (f) and 5 Å (g)  To experimentally explore the electronic structure of 5, we used bilayer NaCl films on coinage metal surfaces to electronically decouple the molecule from the metal surfaces  Before presenting the experimental findings, we summarize the results of our theoretical calculations performed on 5 in the neutral charge state (denoted as 5 0 ) 
We start by performing DFT calculations on 5 0 in the gas phase  Geometry optimization performed at the spin-unrestricted UB3LYP/6-31G level of theory leads to one local minimum, 5OS, the geometry of which corresponds to the open-shell resonance structure of 5 (Fig  , the label OS denotes open-shell) 
The triplet electronic configuration of 5OS is the lowest-energy state, with the openshell singlet configuration 90 meV higher in energy  Geometry optimization performed at the restricted closed-shell RB3LYP/6-31G level reveals two local minima, 5para and 5ortho, the geometries of which (Fig  ) exhibit bond length alternations in line with the presence of a para-or an ortho-QDM moiety, respectively, in the as-indacene core of the closed-shell resonance structures of 5 (Fig  )  
Relative to 5OS in the triplet configuration, 5para and 5ortho are 0 40 and 0 43 eV higher in energy, respectively  Additional DFT results are shown in Supplementary Fig    To gain more accurate insights into the theoretical electronic structure of 5, we performed multireference perturbation theory calculations (Supplementary Fig  ) based on quasi-degenerate second-order n-electron valence state perturbation theory (QD-NEVPT2) 
In so far as the order of the ground and excited states are concerned, the results of QD-NEVPT2 calculations qualitatively match with DFT calculations  For 5OS, the triplet configuration remains the lowest-energy state, with the open-shell singlet configuration 60 meV higher in energy  The energy differences between the open-and closed-shell states are substantially reduced in QD-NEVPT2 calculations, with 5para and 5ortho only 0 11 and 0 21 eV higher in energy, respectively, compared to 5OS in the triplet configuration 
We also performed nucleus-independent chemical shift calculations to probe local aromaticity of 5 in the openand closed-shell states  While 5OS in the triplet configuration exhibits local aromaticity at the terminal benzenoid rings, 5OS in the open-shell singlet configuration, 5para and 5ortho all display antiaromaticity (Supplementary Fig  ) 
The choice of the insulating surface determines the charge state of 5: while 5 adopts neutral charge state on the high work function bilayer NaCl/Au(111) surface (irrespective of its openor closed-shell state, Supplementary Fig  ), 5 exhibits charge bistability between 5 0 and the anionic state 5 -1 on the lower work function bilayer NaCl/Ag(111) and Cu(111) surfaces (Supplementary Figs  ) 
In the main text, we focus on the characterization of 5 on bilayer NaCl/Au(111)  Characterization of charge bistable 5 is reported in Supplementary Figs    We first describe experiments on 5 on bilayer NaCl/Au(111), where 5 exhibits a geometry corresponding to the calculated 5OS geometry, and an open-shell electronic configuration 
We compare the experimental data on this species to calculations on 5OS with a triplet configuration, as theory predicts a triplet ground state for 5OS  For 5OS, the calculated frontier orbitals correspond to the SOMOs ψ1 and ψ2 (Fig  ), whose spin up levels are occupied and the spin down levels are empty 
Figure shows the DFT-calculated bond lengths of 5OS, where the two salient features, namely, the small difference in the bond lengths within each ring and the notably longer bond lengths in the pentagonal rings, agree with the open-shell resonance structure of 5 (Fig  )  Figure shows an AFM image of 5 adsorbed on bilayer NaCl/Au(111) that we assign as 5OS, where the bond-order differences qualitatively correspond to the calculated 5OS geometry (discussed and compared to the closed-shell state below) 
Differential conductance spectra (dI/dV(V), where I and V denote the tunneling current and bias voltage, respectively) acquired on assigned 5OS exhibit two peaks centered at -1 5 V and 1 6 V (Fig  ), which we assign to the positive and negative ion resonances (PIR and NIR), respectively  Figure shows the corresponding STM images acquired at the onset (V = -1 2
V/1 3 V) and the peak (V = -1 5 V/1 6 V) of the ionic resonances  To draw a correspondence between the STM images and the molecular orbital densities, we consider tunneling events as many-body electronic transitions between different charge states of 5OS (Fig  )  Within this framework, the PIR corresponds to transitions between 5 0 and the cationic state 5  
At the onset of the PIR at -1 2 V, an electron can only be detached from the SOMO ψ1 and the corresponding STM image at -1 2 V shows the orbital density of ψ1  Increasing the bias to the peak of the PIR at -1 5 V, it becomes possible to also empty the SOMO ψ2, such that the corresponding STM image shows the superposition of ψ1 and ψ2, that is, |ψ1| 2 + |ψ2| 2 (ref 
)  Similarly, the NIR corresponds to transitions between 5 0 and 5 -1   At the NIR onset of 1 3 V, only electron attachment to ψ2 is energetically possible  At 1 6 V, electron attachment to ψ1 also becomes possible, and the corresponding STM image shows the superposition of ψ1 and ψ2  The observation of the orbital densities of SOMOs, and not the hybridized HOMO and LUMO, proves the open-shell ground state of assigned 5OS 
Measurements of the monoradical species with a doublet ground state are shown in Supplementary Fig    Unexpectedly, another species of 5 was also experimentally observed that exhibited a closedshell ground state  In contrast to 5OS, where the frontier orbitals correspond to the SOMOs ψ1 and ψ2, DFT calculations predict orbitals of different shapes and symmetries for 5para and 5ortho, denoted as α and β and shown in Fig   
For 5ortho, α and β correspond to HOMO and LUMO, respectively  The orbitals are inverted in energy and occupation for 5para, where β is the HOMO and α is the LUMO  Fig  shows an AFM image of 5 that we assign as 5para  We experimentally infer its closed-shell state first by using qualitative bond order discrimination by AFM 
In high-resolution AFM imaging, chemical bonds with higher bond order are imaged brighter (that is, with higher frequency shift Δf) due to stronger repulsive forces, and they appear shorter   In Fig  , we label seven bonds whose bond orders show significant qualitative differences in the calculated 5ortho, 5para (Fig  ) and 5OS (Fig  ) geometries 
In 5para, the bonds b and d exhibit a higher bond order than a and c, respectively  This pattern is reversed for 5ortho, while the bond orders of the bonds a-d are all similar and small for 5OS  Furthermore, in 5para bond f exhibits a higher bond order than e, while in 5ortho and 5OS bonds e and f exhibit similar bond order (because they belong to Clar sextets) 
Finally, the bond labeled g shows a higher bond order in 5para than in 5ortho and 5OS  The AFM image of assigned 5para shown in Fig  indicates higher bond orders of the bonds b, d and f compared to a, c and e, respectively  In addition, the bond g appears almost point-like and with enhanced Δf contrast compared to its neighboring bonds, indicative of a high bond order (see Supplementary Fig  for height-dependent measurements) 
These observations concur with the calculated 5para geometry (Fig  )  Importantly, all these distinguishing bond-order differences are distinctly different in the AFM image of 5OS shown in Fig  , which is consistent with the calculated 5OS geometry (Fig  )  In the AFM images of 5OS (Fig  and Supplementary Fig  ), the bonds a-d at the pentagon apices appear with similar contrast and apparent bond length 
The bonds e and f at one of the terminal benzenoid rings also exhibit similar contrast and apparent bond length, while the central bond g appears longer compared to assigned 5para  Further compelling evidence for the closed-shell state of assigned 5para is obtained by STM and STS  dI/dV(V) spectra acquired on an assigned 5para species exhibit two peaks centered at -1 4 V (PIR) and 1 6 V (NIR) (Fig  ) 
STM images acquired at these biases (Fig  ) show the orbital densities of β (-1 4 V) and α (1 6 V)  First, the observation of α and β as the frontier orbitals of this species, and not the SOMOs, strongly indicates its closed-shell state  Second, consistent with AFM measurements that indicate good correspondence to the calculated 5para geometry, we observe β as the HOMO and α as the LUMO 
For 5ortho, α should be observed as the HOMO and β as the LUMO  We did not observe molecules with the signatures of 5ortho in our experiments  We observed molecules in open-(5OS, Fig  ) and closed-shell (5para, Fig  ) states in similar occurrence after their generation from 6 on the surface  We could also switch individual molecules between open-and closed-shell states as shown in Fig  and Supplementary Fig   
To this end, a change in the adsorption site of a molecule was induced by STM imaging at ionic resonances, which often resulted in movement of the molecule  The example presented in Fig  shows a molecule that was switched from 5para to 5OS and back to 5para  The switching is not directed, that is, we cannot choose which of the two species will be formed when changing the adsorption site, and we observed 5OS and 5para in approximately equal yields upon changing the adsorption site 
The molecule in Fig  is adsorbed on top of a defect that stabilizes its adsorption geometry on bilayer NaCl  At defect-free adsorption sites on bilayer NaCl, that is, without a third layer NaCl island or atomic defects in the vicinity of the molecule, 5 could be stably imaged neither by AFM nor by STM at ionic resonances (Supplementary Fig  ) 
Without changing the adsorption site, the state of 5 (open-or closedshell) never changed, including the experiments on bilayer NaCl/Ag(111) and Cu(111), on which the charge state of 5 could be switched (Supplementary Figs  )  Also on these lower work function surfaces, both open-and closed-shell species were observed for 5 0 and both showed charge bistability between 5 0 (5OS or 5para) and 5 -1 (Supplementary Figs  ) 
The geometrical structure of 5 -1 probed by AFM, and its electronic structure probed by STM imaging at the NIR (corresponding to transitions between 5 -1 and the dianionic state 5 -2 ), are identical within the measurement accuracy for the charged species of both 5OS and 5para  When cycling the charge state of 5 between 5 0 and 5 -1 several times, we always observed the same state (5OS or 5para) when returning to 5 0 , provided the molecule did not move during the charging/discharging process 
Based on our experimental observations we conclude that indeno[1,2-a]fluorene (5), the last unknown indenofluorene isomer, can be stabilized in and switched between an open-shell (5OS) and a closed-shell (5para) state on NaCl  For the former, both DFT and QD-NEVPT2 calculations predict a triplet electronic configuration 
Therefore, 5 can be considered to exhibit the spin-crossover effect, involving magnetic switching between high-spin (5OS) and low-spin (5para) states, coupled with a reversible structural transformation  So far, the spin-crossover effect has mainly only been observed in transition-metal-based coordination compounds with a near-octahedral geometry  
The observation that the switching between open-and closedshell states is related to changes in the adsorption site but is not achieved by charge-state cycling alone, indicates that the NaCl surface and local defects facilitate different electronic configurations of 5 depending on the adsorption site 
Gas-phase QD-NEVPT2 calculations predict that 5OS is the ground state, and the closed-shell 5para and 5ortho states are 0 11 and 0 21 eV higher in energy  The experiments, showing bidirectional switching between 5OS and 5para, indicate that a change in the adsorption site can induce sufficient change in the geometry of 5 (leading to a corresponding change in the ground state electronic configuration) and thus induce switching 
Switching between open-and closed-shell states in 5 does not require the breaking or formation of covalent bonds , but a change of adsorption site on NaCl where the molecule is physisorbed  Our results should have implications for single-molecule devices, capitalizing on the altered electronic and chemical properties of a system in π-diradical open-shell and closed-shell states such as frontier orbital and singlet-triplet gaps, and chemical reactivity 
For possible future applications as a single-molecule switch, it might be possible to also switch between open-and closed-shell states by changing the local electric field, such as by using chargeable adsorbates   Scanning probe microscopy measurements and sample preparation  STM and AFM measurements were performed in a home-built system operating at base pressures below 1×10 -10 mbar and a base temperature of 5 K  Bias voltages are provided with respect to the sample 
All STM, AFM and spectroscopy measurements were performed with carbon monoxide (CO) functionalized tips  AFM measurements were performed in non-contact mode with a qPlus sensor   The sensor was operated in frequency modulation mode with a constant oscillation amplitude of 0 5 Å  STM measurements were performed in constantcurrent mode, AFM measurements were performed in constant-height mode with V = 0 V, and I(V) and Δf(V) spectra were acquired in constant-height mode 
Positive (negative) values of the tip-height offset Δz represent tip approach (retraction) from the STM setpoint  All dI/dV(V) spectra are obtained by numerical differentiation of the corresponding I(V) spectra  STM and AFM images, and spectroscopy curves, were post-processed using Gaussian low-pass filters 
Au(111), Ag(111) and Cu(111) surfaces were cleaned by iterative cycles of sputtering with Ne + ions and annealing up to 800 K  NaCl was thermally evaporated on Au(111), Ag(111) and Cu(111) surfaces held at 323 K, 303 K and 283 K, respectively  This protocol results in the growth of predominantly bilayer (100)-terminated islands, with a minority of trilayer islands 
Sub-monolayer coverage of 6 on surfaces was obtained by flashing an oxidized silicon wafer containing the precursor molecules in front of the cold sample in the microscope  CO molecules for tip functionalization were dosed from the gas phase on the cold sample  Density functional theory calculations  DFT was employed using the PSI4 program package  
All molecules with different charge (neutral and anionic) and electronic (open-and closed-shell) states were independently investigated in the gas phase  The B3LYP exchangecorrelation functional with 6-31G basis set was employed for structural relaxation and singlepoint energy calculations  The convergence criteria were set to 10 −4 eV Å −1 for the total forces and 10 −6 eV for the total energies 
Multireference calculations  Multireference calculations were performed on the DFToptimized geometries using the QD-NEVPT2 level of theory , with three singlet roots and one triplet root included in the state-averaged calculation  A (10,10) active space (that is, 10 electrons in 10 orbitals) was used along with the def2-TZVP basis set  
Increasing either the active space size or expanding the basis set resulted in changes of about 50 meV for relative energies of the singlet and triplet states  These calculations were performed using the ORCA program package   Nucleus-independent chemical shift (NICS) calculations  Isotropic nucleus-independent chemical shift values were evaluated at the centre of each ring using the B3LYP exchangecorrelation functional with def2-TZVP basis set using the Gaussian 16 software package  
Starting materials (reagent grade) were purchased from TCI and Sigma-Aldrich and used without further purification  Reactions were carried out in flame-dried glassware and under an inert atmosphere of purified Ar using Schlenk techniques  Thin-layer chromatography (TLC) was performed on Silica Gel 60 F-254 plates (Merck) 
Column chromatography was performed on silica gel (40-60 µm)  Nuclear magnetic resonance (NMR) spectra were recorded on a Bruker Varian Mercury 300 or Bruker Varian Inova 500 spectrometers  Mass spectrometry (MS) data were recorded in a Bruker Micro-TOF spectrometer  The synthesis of compound 6 was developed following the two-step synthetic route shown in Supplementary Fig  , which is based on the preparation of methylene-bridge polyarenes by means of Pd-catalyzed activation of benzylic C-H bonds  
Supplementary Figure | Synthetic route to obtain compound 6  The complex Pd2(dba)3 (20 mg, 0 02 mmol) was added over a deoxygenated mixture of 1,3-dibromo-2,4-dimethylbenzene (9, 100 mg, 0 38 mmol), boronic acid 10 (178 mg, 1 14 mmol), K2CO3 (314 mg, 2 28 mmol) and XPhos (35 mg, 0 08 mmol) in toluene (1:1, 10 mL), and the resulting mixture was heated at 90 °C for 2 h 
After cooling to room temperature, the solvents were evaporated under reduced pressure  The reaction crude was purified by column chromatography (SiO2; hexane:CH2Cl2 9:1) affording 11 (94 mg, 76%) as a colorless oil  The complex Pd(OAc)2 (7 mg, 0 03 mmol) was added over a deoxygenated mixture of terphenyl 11 (90 mg, 0 27 mmol), K2CO3 (114 mg, 0 83 mmol) and ligand L (26 mg, 0 06 mmol) in NMP (2 mL) 
The resulting mixture was heated at 160 °C for 4 h  After cooling to room temperature, H2O (30 mL) was added, and the mixture was extracted with EtOAc (3x15 mL)  The combined organic extracts were dried over anhydrous Na2SO4, filtered, and evaporated under reduced pressure  The reaction crude was purified by column chromatography (SiO2; hexane:CH2Cl2 9:1) affording compound 6 (8 mg, 11%) as a white solid  in AFM imaging due to their reduced adsorption height compared to the rest of the carbon atoms 
We attribute this observation to the significantly different lattice parameter of Cu(111) (2 57 Å) compared to Au(111) and Ag(111) (2 95 Å and 2 94 Å, respectively) , such that the apical carbon atoms of the pentagonal rings of 5 adsorb on the on-top atomic sites on Au(111) and Ag(111), but not on Cu(111) 
Our speculation is based on a previous study of polymers of 1 on Au(111) by Di Giovannantonio et al  , where both tilted and planar individual units of 1 were observed depending on whether the apical carbon atoms of the pentagonal rings in 1 adsorbed on the on-top or hollow sites of the surface, respectively 
Given the strong molecule-metal interaction, we found no electronic state signatures of 5 on all three metal surfaces  STM set point for AFM images: V = 0  e, Frontier orbital spectrum of 5 -1   In the anionic state, ψ2 becomes doubly occupied and ψ1 is the SOMO  Filled and empty circles denote occupied and empty orbitals, respectively 
For each panel, zero of the energy axis has been aligned to the respective highest-energy occupied orbital 
Context 3: Paper Info

Title: Incorporating Human Path Preferences in Robot Navigation with Minimal Interventions
Publish Date: 16 Mar 2023
Author List: Oriana Peltzer, Dylan Asmar, Mac Schwager, Mykel Kochenderfer

Figure

Hyperplane arrangement of a twodimensional space containing two obstacles (colored in gray) The robot is located inside the pink polytope, surrounded by three adjacent obstacle-free polytopes Each hyperplane on the boundary of the robot's polytope corresponds to one of the nonredundant constraints in eq (4) (b)Graph derived from the hyperplane arrangement The nodes on the graph designate polytopes, and edges designate transitions to adjacent polytopes To estimate the human's preference, the robot updates a posterior over the goal and over which of the graph transitions φ 1 , φ 2 and φ 3 is preferred by the human (c)Example preference defined over the graph The location of the goal is indicated in yellow in the lower right polytope For each node, the outgoing pink arrow designates the edge on the graph corresponding to the preferred transition between polytopes 
Simple, 10 × 10, 8 polytopes (b) Map 2: Office, 10 × 10, 56 polytopes (c) Map 3: Classroom, 20 × 20, 73 polytopes (d) Sampled observations and robot's executed trajectories 
Fig 5: Maps used for simulating the robot navigation problem with path preferences In (d), the heading angles observed are indicated with arrows The goal is indicated with a pink circle, and the orange robot corresponds to the starting location The blue robot follows a policy that accounts for path preference, while the green robot does not The opacity of the robots increases with time 
Map 1 problem setup and example realizations for goal-only (green) and path preference (blue) solution methods The robot starts at the lower left corner of the environment, and the goal of the task (pink circle) is in the upper left area The robot does not know which goal, among 10 options (shown in light blue squares), is the correct goal The human provides noisy observations, indicated by arrows, at each iteration The green robot selects actions according to the goal-only baseline, and the blue robot uses our proposed method to infer path preferences The polytopes composing G are drawn in blue Probability of correct goal WLPHVWHS +J (c) Entropy of goal distribution g 
Fig  6: Probability of the correct goal, fig 6b, and entropy of the goal belief distribution P (g), fig 6c, for the same problem setup, fig 6a In this problem instance, the human's preference is to go to the goal by passing on the right side of the obstacle Results are averaged over 50 runs and the area filled represents one standard deviation above and below the mean value The goal-only baseline shows an over-confident prediction (shown by the strong reduction in belief entropy) that the correct goal is less likely, making it more difficult to reach the correct goal compared to a method that accounts for path preference 
Success rates in the simple environment (Map 1) The results are averaged over 6 randomly sampled problem instances (start location, goal location, and goal possibilities), and over 50 runs per problem instance ∆T is the number of time steps separating two consecutive human inputs The robot's mission time is Tmax = 30 time steps We selected γ h = 1 5, corresponding to relatively noisy human inputs and making the problem more difficult to solve for the robot 
Computation times for Goal Only and Path Preference methods on Map 1 (fig 5a),Map 2 (fig 5b), and Map 3 (fig 5c),averaged over 100 runs with randomly sampled problem instances The 95 % confidence interval is provided with the mean We evaluate computation time at the first iteration of each run (where the search depth takes on its highest value Tmax) 

abstract

Robots that can effectively understand human intentions from actions are crucial for successful human-robot collaboration  In this work, we address the challenge of a robot navigating towards an unknown goal while also accounting for a human's preference for a particular path in the presence of obstacles 
This problem is particularly challenging when both the goal and path preference are unknown a priori  To overcome this challenge, we propose a method for encoding and inferring path preference online using a partitioning of the space into polytopes  Our approach enables joint inference over the goal and path preference using a stochastic observation model for the human 
We evaluate our method on an unknown-goal navigation problem with sparse human interventions, and find that it outperforms baseline approaches as the human's inputs become increasingly sparse  We find that the time required to update the robot's belief does not increase with the complexity of the environment, which makes our method suitable for online applications 

INTRODUCTION

Collaboration between humans and robots has become increasingly important and one key aspect of this collaboration is the ability for robots to adapt to human decisions  In many scenarios, such as a robot navigating through a busy room to deliver an item, it is important for the robot to take into account human preferences 
For instance, humans may prefer a specific path that would allow their colleagues to notice the item being delivered, but this preference may change dynamically based on various factors such as changes in the environment or unforeseen circumstances  While some preferences can be incorporated into the path-planning process, accommodating dynamic user preferences in real-time remains challenging 
In this paper, we propose a way to enable robots to adapt to human preferences dynamically by leveraging real-time feedback to inform decision-making  In this work, we tackle the problem of robot navigation in which the robot cannot observe the goal or the preferred path to the goal, but must make navigation decisions that are influenced by humans through recommended actions 
Prior work has explored how to adapt to a human's preference through feedback, but such approaches often require a high level of intervention, which can be time-consuming and impractical in real-world scenarios  To optimize the use of human input and quickly infer the human's preference, Fig  : An autonomous robot navigates in a simulated classroom towards a goal location (pink circle) 
At the start of its mission, it receives direction indications (arrows) from a human that indicate which path it should take to get to the goal  In this scenario, the human wants the robot to go around the desks on the right side of the classroom  A robot that does not reason over path preferences (green) will take the shortest path to the goal regardless of the human's input 
Our method (blue) infers the human's path preference from these indications and adapts to their recommendations  we propose an approach that leverages probabilistic representations of human preference and incorporates real-time feedback  Previous research by Bajcsy et al  considered an online adaptation problem in a manipulation task, where the person can apply forces to the robot to indicate their preferences 
By allowing the robot to continue its task while taking into account a probabilistic representation of human preference, their approach does not require frequent inputs  Building on this idea, we adopt a similar approach to adapt to a human's preference in the context of a robot autonomously navigating through a known environment, such as a cluttered office space 
Specifically, we focus on allowing the human to influence the robot's trajectory with respect to obstacles, by providing guidance on preferred routes or paths, while the robot continues to execute its task  Paths can be represented using homotopy classes   However, homotopies can pose computational challenges when used to encode and infer human preferences 
When the robot maintains a belief over homotopy classes, the inference problem can become exponentially complex with the number of obstacles in the space  Additionally, when the goal is unknown, the number of variables increases with the number of candidate destinations  This complexity can render the decision-making problem intractable 
Our solution is to encode path preference based on a partitioning of the environment into polytopes   This representation allows path preferences to be expressed as sets of preferred transitions between adjacent polytopes  Paths belonging to different homotopy classes correspond to different sequences of transitions 
By leveraging conditional independence assumptions, we can make the Bayesian inference problem tractable  These assumptions exploit the fact that human actions provide information about the path in a piece-wise manner  For example, indicating a preference for navigating around a particular obstacle only provides information about the local area and not the entire path 
Finally, after updating its belief representation over the human's preference, the robot can adapt to indications by replanning online  Our contributions are as follows  • We formulate the human-robot collaboration problem as a Partially Observable Markov Decision Process (POMDP) where both the goal of the task and the human's path preference are unknown random variables 
• We propose an encoding of a human's path preference using a partitioning of the environment into polytopes, along with conditional independence assumptions that make the Bayesian inference problem tractable to infer the task goal and path preference online  • Through simulations in two environments of different sizes and complexity, we show that our method is effective for solving problems where the robot must reach a goal that is unknown a-priori while simultaneously adapting to a human's indications 
Our method shows higher success rates compared to baseline approaches when the human inputs are sparse  Our approach enables a robot to make effective navigation decisions in collaboration with a human, even when the goal and path preference are not known in advance, and with minimal human input  In recent years, there has been a growing interest in shared autonomy and interactive systems, where humans and robots work together to accomplish tasks 
Several approaches have been proposed to address the challenge of enabling effective collaboration between human and robot agents while still achieving high task performance  Losey et al  and Jeon, Losey, and Sadigh propose a framework where a human operator is given control of a task-relevant latent action space while an autonomous system handles the rest 
Dragan and Srinivasa present a formalism for arbitrating between a user's input and a robot's policy when both human and robot share control of the same action space  Cognetti et al  [7] provide a method for real-time modifications of a path,   



ground_truth: Open-shell π-diradical state and closed-shell state with a para-quinodimethane moiety.

prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: What is the potential of SNNs in modeling the visual system?

Read the following text and answer briefly.

Context 2: Paper Info

Title: Deep Spiking Neural Networks with High Representation Similarity Model Visual Pathways of Macaque and Mouse
Publish Date: 22 May 2023
Author List: Zhengyu Ma (from Department of Networked Intelligence, Peng Cheng Laboratory), Yu Liutao (from Department of Networked Intelligence, Peng Cheng Laboratory), Huihui Zhou (from Department of Networked Intelligence, Peng Cheng Laboratory), Allen Brain
Author Affiliation: CORNet-S ConvNeXt-Tiny ConvNeXt-Small EfficientNet, AlexNet RegNetY, ResNet34 ConvNeXt-Base CORNetSEW, ResNet8 ResNet101 SEW-ResNet18 ViT-L, GoogLeNet SEW-ResNet34 SEW-ResNet8 Wide

Figure

Figure 1: To conduct neural representation similarity experiments, we apply three similarity metrics to a layer-by-layer comparison between the responses of models and the neural activities of visual cortex 
Figure 2: For three datasets and three similarity metrics, each point indicates the final representation similarity score of a model Each pair of SEW ResNet and ResNet with the same depth are linked by a gray solid line In almost all conditions, SEW ResNet outperforms ResNet by a large margin 
Figure3: For three datasets and three similarity metrics, we plot the trajectories of similarity score with model layer depth The models are divided into two groups: ResNet and SEW ResNet The normalized layer depth ranges from 0 (the first layer) to 1 (the last layer) Because the depths of models are not the same, we first discretize the normalized depth into 50 bins, and then apply the cubic spline interpolation to the scores of each model, yielding the smooth trajectories shown in the plot The fine, semitransparent lines are the trajectories of each model The thick lines are the average trajectories among each group 
Figure 5: For Macaque-Synthetic dataset, trajectories of similarity score with model layer depth are plotted The models are divided into two groups: ViT and CNN&SNN The normalized layer depth ranges from 0 (the first layer) to 1 (the last layer) The calculation and plotting of the trajectories are the same as Figure 3 
Figure6: The basic block of SpikingMobileNet "PW CONV" is the pointwise convolution and "DW CONV" is the depthwise convolution "SN" is the spiking neuron 
Figure 7: Overall model rankings of the similarity scores on Allen Brain mouse dataset The similarity scores of CNNs, SNNs and vision transformers are shown by blue, green and orange bars, respectively 
Figure 9: Overall model rankings of the similarity scores on Macaque-Synthetic dataset 
Figure 10: The Spearman's rank correlation between the overall model rankings of different metrics There is a strong correlation between SVCCA and TSVD-Reg, but RSA has weaker correlations with them 
The correlation between the similarity scores and the model depth r is Spearman's rank correlation coefficient "-" indicates that there is no significant correlation 
Architectures of SNNs "sn" denotes the spiking neuron "g = 32" denotes the grouped convolutions with 32 groups The hyper-parameters of the spike-element-wise block are shown in the brackets with the number of stacked blocks outside 

abstract

Deep artificial neural networks (ANNs) play a major role in modeling the visual pathways of primate and rodent  However, they highly simplify the computational properties of neurons compared to their biological counterparts  Instead, Spiking Neural Networks (SNNs) are more biologically plausible models since spiking neurons encode information with time sequences of spikes, just like biological neurons do 
However, there is a lack of studies on visual pathways with deep SNNs models  In this study, we model the visual cortex with deep SNNs for the first time, and also with a wide range of state-of-the-art deep CNNs and ViTs for comparison  Using three similarity metrics, we conduct neural representation similarity experiments on three neural datasets collected from two species under three types of stimuli 
Based on extensive similarity analyses, we further investigate the functional hierarchy and mechanisms across species  Almost all similarity scores of SNNs are higher than their counterparts of CNNs with an average of 6 6%  Depths of the layers with the highest similarity scores exhibit little differences across mouse cortical regions, but vary significantly across macaque regions, suggesting that the visual processing structure of mice is more regionally homogeneous than that of macaques 
Besides, the multi-branch structures observed in some top mouse brain-like neural networks provide computational evidence of parallel processing streams in mice, and the different performance in fitting macaque neural representations under different stimuli exhibits the functional specialization of information processing in macaques 
Taken together, our study demonstrates that SNNs could serve as promising candidates to better model and explain the functional hierarchy and mechanisms of the visual system  Originally, the prototype of deep neural networks is inspired by the biological vision system   To date, deep neural networks not only occupy an unassailable position in the field of computer vision , but also become better models of the biological visual cortex compared to traditional models in the neuroscience community (Khaligh-Razavi and Kriegeskorte 2014;  
They have been successful at predicting the neural responses in primate visual cortex, matching the hierarchy of ventral visual stream (Güc ¸lü and van Gerven 2015; , and even controlling neural activity   Moreover, as training paradigms of mice and techniques for collecting neural activity (de Vries et al  2020) have been greatly improved, there is a strong interest in exploring mouse visual cortex 
Deep neural networks also play an important role in revealing the functional mechanisms and structures of mouse visual cortex   Compared to biological networks, Artificial Neural Networks discard the complexity of neurons   Spiking Neural Networks, incorporating the concept of time and spikes, are more biologically plausible models  
To be more specific, because of their capabilities of encoding information with spikes, capturing the dynamics of biological neurons, and extracting spatio-temporal features, deep SNNs are highly possible to yield brain-like representations )  However, deep SNNs have not been employed to model visual cortex due to the immaturity of training algorithms 
Recently, a state-ofthe-art directly trained deep SNN , makes it possible to use deep SNNs as visual cortex models  Contributions  In this work, we conduct large-scale neural representation similarity experiments on SNNs and other high-performing deep neural networks to study the brain's visual processing mechanisms, with three datasets and three similarity metrics (Figure ) 
Specifically, to the best of our knowledge, we are the first to use deep SNNs to fit complex biological neural representations and explore the biological visual cortex  We summarize our main contributions in four points as follows  • We find that SNNs outperform their counterparts of CNNs with the same depth and almost the same architectures in almost all experiments 
In addition, even with very different depths and architectures, SNNs can achieve top performance in most conditions  • By making a more direct comparison between macaques and mice for the first time, we reveal the differences in the visual pathways across the two species in terms of the homogeneity of visual regions and the increases of receptive field sizes across cortical visual pathways, which is consistent with previous physiological work 
• The multi-branch structures in neural networks benefit neural representation similarity to mouse visual cortex, providing computational evidence that parallel information processing streams are widespread between cortical regions in the mouse visual system  • Comparing the results of two macaque neural datasets under different stimuli, we reveal that the macaque vision system may have functional specialization for processing human faces and other natural scenes 
Altogether, as the first work to apply deep SNNs to fit neural representations, we shed light on visual processing mechanisms in both macaques and mice, demonstrating the potential of SNNs as a novel and powerful tool for research on the visual system  Our codes and appendix are available at https://github com/Grasshlw/SNN-Neural-Similarity 
There are plenty of computational models of macaque and mouse visual systems for exploring the visual processing mechanisms recently  We summarize some of the outstanding work in the following  The network models of macaque visual system  In the early days, studies basically used simple feedforward neural networks as the models of the macaque visual system (Khaligh-Razavi and Kriegeskorte 2014;  
Recently, some bio-inspired or more complex models achieved better performance in fitting the neural representations of macaque visual cortex   proposed a brainlike shallow CNN with recurrent connections to better match the macaque ventral visual stream  By mimicking the primary stage of the primate visual system, VOneNets ) performed more robustly in image recognition while better simulating macaque V1 
Moreover, the representations learned by unsupervised neural networks ) also effectively matched the neural activity of macaque ventral visual stream  Although the above work developed many bio-inspired structures, the networks are still traditional ANNs in nature  Our work introduces deep SNNs for the first time to explore the visual processing mechanisms of macaque visual system 
The network models of mouse visual system  Largescale mouse neural dataset provided an experimental basis for model studies of mouse visual system (de Vries et al  2020;   conducted comparisons between the representations of mouse visual cortex and the VGG16 trained on the Im-ageNet dataset  In , they developed a single neural network to model both the dorsal and ventral pathways with showing the functional specializations 
What's more, a large survey of advanced deep networks ) revealed some hierarchy and functional properties of mice  Similar to the studies of macaque visual system, deep SNNs have never been used to model the mouse visual system  In this work, we not only use SNNs as one of the candidates to fit the representations of mouse visual cortex, but also conduct direct comparisons between macaques and mice to further investigate the functional hierarchy and mechanisms of the two species 
Our work is conducted with three neural datasets  These datasets are recorded from two species under three types of stimuli  More specifically, there are neural responses of mouse visual cortex to natural scene stimuli, and responses of macaque visual cortex to face image and synthetic image stimuli  Allen Brain mouse dataset 
It is part of the Allen Brain Observatory Visual Coding dataset ) col-lected using Neuropixel probes from 6 regions simultaneously in mouse visual cortex  Compared to two-photon calcium imaging, Neuropixel probes simultaneously record the spikes across many cortical regions with high temporal resolution 
In these experiments, mice are presented with 118 250-ms natural scene stimuli in random orders for 50 times  Hundreds to thousands of neurons are recorded for each brain region  To get the stable neurons, we first concatenate the neural responses (average number of spikes in 10-ms bins across time) under 118 images for each neuron, and then preserve the neurons whose split-half reliability across 50 trials reaches at least 0 8 
Macaque-Face dataset  This dataset ) is composed of neural responses of 159 neurons in the macaque anterior medial (AM) face patch under 2,100 real face stimuli, recorded with Tungsten electrodes  For this dataset, we compute the average number of spikes in a time window of 50-350ms after stimulus onset and exclude eleven neurons with noisy responses by assessing the neurons' noise ceiling 
The details of the preprocessing procedure are the same as   Macaque-Synthetic dataset  This dataset is also about macaque neural responses which are recorded by electrodes under 3,200 synthetic image stimuli, and used for neural prediction in the initial version of Brain-Score   The image stimuli are generated by adding a 2D projection of a 3D object model to a natural background 
The objects consist of eight categories, each with eight subclasses  The position, pose, and size of each object are randomly selected  88 neurons of V4 and 168 neurons of IT are recorded  The neural responses are preprocessed to the form of average firing rate and can be downloaded from Brain-Score  Since the core visual function of macaque and mouse visual cortex is to recognize objects, the basic premise of model selection is that the model has good performance on object recognition tasks (e g 
classification on ImageNet)  Based on this premise, we employ 12 SNNs, 43 CNNs, and 26 vision transformers, all of which are pretrained on the Ima-geNet dataset and perform well in the classification task  As for SNNs, we use SEW ResNet as the base model, which is the deepest and SOTA directly trained SNN  
Furthermore, by combining the residual block used in SEW ResNet and the hierarchy of the visual cortex, we build several new SNNs and train them on the ImageNet using SpikingJelly ) (see Appendix A for model structures and the details of model training)  As for CNNs and vision transformers, we use 44 models from the Torchvision model zoo , 22 models from the Timm model zoo ) and 3 models from the brain-like CNNs, CORnet family ) 
In the feature extraction procedures of all models, we feed the same set of images used in biological experiments to the pretrained models and obtain features from all chosen layers  Different from CNNs and vision transformers, the features of SNNs are spikes in multiple time steps  To obtain the representation similarity between biological visual cortex and computational models, we apply three similarity metrics to computing similarity scores: representational similarity analysis (RSA) , regression-based encoding method and singular vector canonical correlation analysis (SVCCA)  
RSA has already been widely used to analyze neural representations of a model and a brain to different stimuli at the population level, while the regression-based encoding method directly fits the model features to neural activity data  SVCCA is originally proposed to compare features of deep neural networks, and then Buice 2019) used it to compare representation matrices from mouse visual cortex and DNNs, which demonstrated its effectiveness 
With the same model and same cortical region, we use these metrics for a layer-by-layer comparison to compute the similarity scores  The maximum similarity score across layers for a given cortical region is considered to be the level of representation similarity between the model and the cortical region 
Finally, in a given dataset, we take the average score of all cortical regions as the final similarity score for each model, which gives the overall model rankings  The implementation of each similarity metric is as follows  RSA  For two response matrices R ∈ R n×m from each layer of models and each cortical region, where n is the number of units/neurons and m is the number of stimuli, we calculate the representational similarity between the responses to each pair of image stimuli using the Pearson correlation coefficient r, yielding two representational dissimilarity matrices (RDM ∈ R m×m , where each element is the correlation distance 1 − r) 
Then, the Spearman rank correlation coefficient between the flattened upper triangles of these two matrices is the metric score  Regression-Based Encoding Method  Firstly, we run truncated singular value decomposition (TSVD) to reduce the feature dimension of model layers to 40  Secondly, the features after dimensionality reduction are fitted to the representations of each neuron by ridge regression 
Finally, we compute the Pearson correlation coefficient between the predicted and ground-truth representations of each neuron and take the mean of all correlation coefficients as the metric score  More specifically, we apply leave-one-out crossvalidation to obtain predicted representations of each neuron 
For simplicity, we name this method 'TSVD-Reg'  SVCCA  For both the responses of model layers and cortical regions, we use TSVD to reduce the dimension of unit/neuron to 40, yielding two reduced representation matrices  Then we apply canonical correlation analysis (CCA) to these two matrices to obtain a vector of correlation coefficients (the length of the vector is 40) 
The metric score is the mean of the vector  Because of the invariance of CCA to affine transformations , in this procedure, we only need to ensure that the stimulus dimension is consistent and aligned, even if the unit/neuron dimension is different  Dimensionality reduction plays an important role in this method to make the number of model features comparable to the number of neurons in cortical regions, since the former usually far exceeds the latter 
In addition, dimensionality reduction helps to determine which features are important to the original data, while CCA suffers in important feature detection  Using just CCA performs badly, which has been proven by   To check how similar the models are to the visual cortex's mechanisms in visual processing, we rank the final similarity scores of all models and conduct comparisons among three types of models (CNNs, SNNs, and vision transformers) 
Specially, we focus on comparing SNN (SEW ResNet) and CNN (ResNet) with the same depth and almost the same architectures (Figure )  The final similarity score of a model is the average similarity score across all cortical regions  (The overall rankings can be found in Appendix B and the comparisons among three types of models are shown in Appendix C )
Allen brain mouse dataset  No single model achieves the highest final similarity scores with all three metrics  For a fair comparison, we apply the paired t-test to SEW ResNet and ResNet with the same depth  For all three metrics, SEW ResNet performs better than ResNet by a large margin (t = 5 857, p = 0 004; t = 7 666, p = 0 002; t = 7 592, p = 0 002) 1   1 The results of the three similarity metrics are separated by semicolons, in the order of SVCCA, TSVD-Reg, and RSA 
Other Macaque-Face dataset  For both SVCCA and TSVD-Reg, Wide-SEW-ResNet14 and Wide-SEW-ResNet8 achieve the first and second highest final similarity scores respectively  But for RSA, TNT-S and Inception-ResNet-V2 take their place and outperform other models by a large margin  As for SEW ResNet and ResNet, the former performs significantly better than the latter for both SVCCA and TSVD-Reg (t = 8 195, p = 0 001; t = 7 528, p = 0 002) 
However, the difference is not significant for RSA (t = 1 117, p = 0 327)  Specifically, the similarity score of SEW ResNet152 is only slightly higher than that of ResNet152, and at the depth of 50 and 101, SEW ResNet's scores are lower than ResNet's  Macaque-Synthetic dataset  Similar to the results of Allen Brain dataset, no model performs best for all three metrics 
SEW ResNet performs moderately better than ResNet (t = 3 354, p = 0 028; t = 3 824, p = 0 019; t = 2 343, p = 0 079)  The only contrary is that SEW ResNet18 performs worse than ResNet18 for RSA  Further, to check the details of comparison between the SNNs and their CNN counterparts, we analyze the trajectories of similarity score across model layers (Figure ) 
As for ResNet and SEW ResNet with the same depth, the trends of their similarities across model layers are almost the same, but the former's trajectory is generally below the latter's  In other words, the similarity scores of SEW ResNet are higher than those of ResNet at almost all layers  Taken together, the results suggest that when the overall results that appear below also correspond to the three metrics in this order, unless the correspondence is stated in the text 
architectures and depth are the same, SNNs with spiking neurons perform consistently better than their counterparts of CNNs with an average increase of 6 6%  Besides, SEW ResNet14 also outperforms the brain-like recurrent CNN, CORnet-S, with the same number of layers (see more details in Appendix B)  Two properties of SNNs might contribute to the higher similarity scores 
On the one hand, IF neurons are the basic neurons of spiking neural networks  The IF neuron uses several differential equations to roughly approximate the membrane potential dynamics of biological neurons, which provides a more biologically plausible spike mechanism for the network  On the other hand, the spiking neural network is able to capture the temporal features by incorporating both time and binary signals, just like the biological visual system during information processing 
To figure out the distinctions in the functional hierarchy between macaques and mice, for each cortical region, we obtain the normalized depth of the layer that achieves the highest similarity score in each model  Then, we divide models (excluding vision transformers) into two groups based on their depths and conduct investigations on these two groups separately 
A nonparametric ANOVA is applied to each group for testing whether layer depths change significantly across cortical regions  For mouse visual cortex (Figure (a)), taking the deep model group as an example, ANOVA shows overall significant changes in depth across cortical regions for TSVD-Reg and RSA (Friedman's χ 2 = 49 169,
p = 2 0 × 10 −9 ; χ 2 = 19 455, p = 0 002)  But there is no significant change for SVCCA (χ 2 = 8 689, p = 0 122)  According to these results, the differences in depth across regions are indeterminacy and irregular  Meanwhile, the trends of layer depth between some regions contradict the hierarchy observed in physiological experiments of mice (those between VISp and VISrl for TSVD-Reg and between VISal and VISpm for RSA) 
However, for macaque visual cortex (Figure (b)), there are significant differences (t = −5 451, p = 6 5 × 10 −6 ; t = −8 312, p = 2 8 × 10 −9 ; t = −3 782, p = 6 9 × 10 −4 , also taking the deep model group as an example) between V4 and IT, and the trend is consistent with the information processing hierarchy in primate visual cortex 
The comparative analyses of the best layer depths of the shallow and deep model groups also exhibit the differences between macaques and mice  For mouse visual cortex, the best layer depths of shallow models are significantly higher than those of deep models  Compared to deep models, most shallow models achieve the top similarity scores in intermediate and even later layers 
Differently, for macaque visual cortex, the depth of models has little effect on the depth of the most similar layer  What's more, we find that the most similar layer of mouse visual cortex always occurs after the 28 × 28 feature map is downsampled to 14 × 14, which leads to the layer depths' difference between shallow and deep models 
Nevertheless, the best layer of macaque IT appears in the last part of networks, where the feature map has been downsampled more times  In summary, our results might reveal two distinctions in the functional hierarchy between macaques and mice  First, there is a distinct functional hierarchical structure of macaque ventral visual pathway, while there might be no clear sequential functional hierarchy in mouse visual cortex 
One explanation is that the mouse visual cortex is organized into a parallel structure and the function of mouse cortical regions are more generalized and homogeneous than those of macaques  Another possibility would be that even though the sequential relations exist among mouse cortical regions as proposed in anatomical and physiological work, they are too weak for the current deep neural networks to capture 
Additionally, mice perform more complex visual tasks than expected with a limited brain capacity   Consequently, the neural responses of mouse visual cortex may contain more information not related to object recognition that neural networks focus on  Secondly, it is well known that the units in the neural networks get larger receptive fields after downsampling, and through the analyses of differences between two groups of models based on depth, we find the feature map of the best layer for mouse is downsampled fewer times than that for macaque 
Based on these results, we provide computational evidence that the increased ratio of the receptive field size in cortical regions across the mouse visual pathway is smaller than those across the macaque visual pathways, which echoes some physio- Macaque-Face dataset --- Table : The correlation between the similarity scores and the number of parameters 
r is Spearman's rank correlation coefficient  "-" indicates that there is no significant correlation  To explore the processing mechanisms in the visual cortex of macaques and mice, we investigate the model properties from the whole to the details  As shown in Table and 2, we first measure the correlation between the similarity scores and the sizes (i e  the number of trainable parameters and the depth) of network models 
For Allen Brain mouse dataset, there are significant negative correlations between the similarity scores and the number of parameters for three metrics while there is no correlation with the depth  Conversely, for the two macaque neural datasets, the similarity scores are highly correlated with the depth of networks, but not with the number of parameters 
Specifically, there is a positive correlation for Macaque-Face dataset while a negative correlation for Macaque-Synthetic dataset  (We also apply the linear regression to analyze the correlation between the similarity scores and the model size  The results are consistent with Spearman's rank correlation and are shown in Appendix E) 
Based on these results, we further investigate more detailed properties of neural networks to explain the processing mechanisms in the visual cortex  For the mouse dataset, on the one hand, the best layer depths show non-significant changes across the mouse cortical regions as mentioned in the previous section 
On the other hand, the similarity scores of the mouse dataset are only correlated with the number of model parameters but not with the depth of models  It calls into the question whether any detailed structures in the neural networks help to reduce the number of parameters and improve its similarity to mouse visual cortex 
Therefore, we explore the commonalities between models that have the top 20% representation similarities (see Appendix D) for Allen Brain dataset  As expected, the top models contain similar structures, such as fire module, inception module, and depthwise separable convolution  All these structures essentially process information through multiple branches/channels and then integrate the features from each branch 
The models with this type of structure outperform other models (t = 2 411, p = 0 024; t = 3 030, p = 0 007; t = 1 174, p = 0 247)  Moreover, we apply the depthwise separable convolution to SNNs, which yields a positive effect  The representation similarity of Spiking-MobileNet is higher than SEW-ResNet50 with a similar depth (+0 8%; +3 9%; +12 1%) 
In fact, some studies using multiple pathways simulate the functions of mouse visual cortex to some extent   Our results further suggest that not only the mouse visual cortex might be an organization of parallel structures, but also there are extensive parallel information processing streams between each pair of cortical regions  
For the two macaque datasets with different stimuli, not only are the model rankings significantly different, but also the correlations between the similarity scores and the model depth are totally opposite  These results corroborate the following two processing mechanisms in macaques: the ventral visual stream of primate visual cortex possesses canonical coding principles at different stages; the brain exhibits a high degree of functional specialization, such as the visual recognition of faces and other objects, which is reflected in the different neural responses of the corresponding region (although the face patch AM is a sub-network of IT, they differ in the neural representations) 
Besides, as shown in Figure , The calculation and plotting of the trajectories are the same as Figure   the similarity scores of vision transformers reach the maximum in the early layers and then decrease  Differently, the scores of CNNs and SNNs keep trending upwards, reaching the maximum in almost the last layer 
On the other hand, Appendix C shows that vision transformers perform well in Macaque-Face dataset but poorly in Macaque-Synthetic dataset  Considering the features extraction mechanism of vision transformers, it divides the image into several patches and encodes each patch as well as their internal relation by self-attention 
This mechanism is effective for face images that are full of useful information  However, the synthetic image consists of a central target object and a naturalistic background  When vision transformers are fed with this type of stimuli, premature integration of global information can lead to model representations containing noise from the unrelated background 
What's more, when we take all models with the top 20% representation similarities as a whole for analyses, as described in the above paragraph, the properties that enable networks to achieve higher neural similarity are not yet clear  Taken together, the computational mechanism of the better models may reveal core processing divergence to different types of stimuli in the visual cortex 
In this work, we take large-scale neural representation similarity experiments as a basis, aided by analyses of the similarities across models and the visual cortical regions  Compared to other work, we introduce SNNs in the similarity analyses with biological neural responses for the first time, showing that SNNs achieve higher similarity scores than CNNs that have the same depth and almost the same architectures 
As analyzed in Section 3 1, two properties of SNNs might serve as the explanations for their high similarity scores  The subsequent analyses of the models' simulation performance and structures indicate significant differences in functional hierarchies between macaque and mouse visual cortex  As for macaques, we observed a clear sequential hi-erarchy 
However, as for mouse visual cortex, some work ) exhibits that the trend of the model feature complexity roughly matches the processing hierarchy, but other work suggests that the cortex ) is organized into a parallel structure  Our results are more supportive of the latter  Furthermore, we provide computational evidence not only that the increased ratio of the receptive field size in cortical regions across the mouse visual pathway is smaller than those across the macaque visual pathway, but also that there may be multiple pathways with parallel processing streams between mouse cortical regions 
Our results also clearly reveal that the processing mechanisms of macaque visual cortex differ to various stimuli  These findings provide us with new insights into the visual processing mechanisms of macaque and mouse, which are the two species that dominate the research of biological vision systems and differ considerably from each other 
Compared to CNNs, the study of task-driven deep SNNs is just in its initial state  Although we demonstrate that SNNs outperform their counterparts of CNNs, SNNs exhibit similar properties as CNNs in the further analyses  In this work, we only build several new SNNs by taking the hints from the biological visual hierarchy, while many well-established structures and learning algorithms in CNNs have not been applied to SNNs yet 
In addition, the neural datasets used in our experiments are all collected under static image stimuli, lacking rich dynamic information to some certain, which may not fully exploit the properties of SNNs  Given that SNNs perform well in the current experiments, we hope to explore more potential of SNNs in future work 
In conclusion, as more biologically plausible neural networks, SNNs may serve as a shortcut to explore the biological visual cortex  With studies on various aspects of SNNs, such as model architectures, learning algorithms, processing mechanisms, and neural coding methods, it's highly promising to better explain the sophisticated, complex, and diverse vision systems in the future 

Implementation Details of SNNs Spiking Neuron Model

For all SNNs, we use the Integrate-and-Fire (IF) model as the spiking neuron model, which acts as the activation layer in neural networks  As mentioned in , V t , X t and S t denote the state (membrane voltage), input (current) and output (spike) of the spiking neuron model respectively at time-step t, and the dynamics of the IF model can be described as follows:
(1) (2) (3) While V t is the membrane voltage after the trigger of a spike, H t is also the membrane voltage, but after charging and before a spike firing  Θ(x) is the unit step function, so S t equals 1 when H t is greater than or equal to the threshold voltage V thresh and 0 otherwise  Meanwhile, when a spike fires, V t is reset to V reset  
Here, we set V thresh = 1 and V reset = 0  In addition, because Θ(x) is non-differentiable at 0, the surrogate gradient method is applied to approximate the derivative function during back-propagation  Here, we use the inverse tangent function as the surrogate gradient function and the derivative function is
(5) In our experiments on SNNs, we not only use SEW ResNet proposed by ), but also build several new SNNs  On the one hand, we improve the spike-elementwise block in SEW ResNet with new architectures referring to studies on ResNet , as shown in Table   On the other hand, as the multi-branch structures in CNNs increase neural representation similarity to mouse visual cortex, we use depthwise separable convolutions and follow the overall architecture of MobileNetV2 to build the SpikingMobileNet, the basic block of which is shown in Figure  
Our implementation is based on SpikingJelly , an open-source framework of deep SNN  We use the ImageNet dataset to pre-train the new SNNs  Following the settings for training SEW ResNet , we train the models for 320 epochs on 8 GPUs (NVIDIA V100), using SGD with a mini-batch size of 32  The momentum is 0 9 and the weight decay is 0  The initial learning rate is 0 1 and we decay it with a cosine annealing, where the maximum number of iterations is the same as the number of epochs 
For all SNNs, we set the simulation duration T = 4 

Overall model rankings

The results of model rankings are shown in Figure , 8 and 9  We also apply the Spearman's rank correlation to the overall model rankings of different metrics, which is shown in Figure  

Score Comparisons among Model Groups

We conduct comparisons of similarity scores among CNNs, SNNs, and vision transformers  The results are shown in Figure  

Overall CNN rankings

The results of CNN rankings are shown in Figure , 13 and 14 

Correlations between the Model Sizes and the Similarity Scores

The results of linear regression to model sizes and the similarity scores are shown in Figure , 16 and 17 

The ImageNet Accuracy and the Similarity Scores

The results are shown in Figure  
Context 3:    Fig  : We model the intent inference problem with the above diagram 
At each step in time, the robot receives an observation ot from the human conditioned on its current location st, the intended goal g, and the human's path preference θ  The robot updates its belief over g and θ and transitions to a next location st+1  while Hagenow et al  present a method that allows an outside agent to modify key robot state variables and blends the changes with the original control 
However, a common challenge of these approaches is the high level of intervention required from humans  Best and Fitch propose a method for predicting an agent's intended trajectory from observations  Rather than maintaining a belief over the agent's future path, they infer the agent's intended goal among a set of candidate locations at the boundary of the space 
This approach provides information on where the agent is heading and generates a distribution of candidate future trajectories for the agent  Inferring the goal of the task among a discrete set of candidates is also relevant to the area of shared autonomy  Javdani, Srinivasa, and Bagnell propose a formalism for shared control of a robotic arm, where the robot must assist the human in picking up an object but needs to infer which object the human has chosen from joystick inputs 
Planning with homotopy class constraints is useful in problems where the robot's requirements are given with respect to obstacles, and Yi, Goodrich, and Seppi consider topological constraints provided by human operators  Bhattacharya propose an efficient algorithm for solving pathplanning problems under homotopic constraints 
However, the number of homotopy classes for a given problem can be infinite, and as the robot changes location and updates its representation of the world, carrying out inference over homotopy classes in a dynamic environment requires recomputing the set of homotopies at every iteration, making the belief update challenging 
Prior work has addressed the challenge of shared autonomy by considering how robots can infer a human's intended goal, or how they can infer the preferred path to a goal  However, we argue that inferring the goal and the path as separate problems can lead to over-confidence in incorrect beliefs about the user's preferences 
To illustrate this point, consider the following scenario: a robot and a human are collaborating to move an object from one end of a room to Fig  : Using the hyperplanes composing the H-representation of each obstacle, we construct a hyperplane arrangement of the obstacle-free space (a)  We define the human's preference for the robot's one step action choices as the posterior distribution (given all human input up to that point) over transitions from the current to the neighboring polytopes, i e  edges on the graph 
Each time the robot transitions to a new polytope, the set of neighbor polytopes and the distribution over human preferences are updated  another, but there is an obstacle in the way  The human would like the robot to take a path around the obstacle on the left, even though the goal is on the right  If the robot only infers the goal from the human's inputs, it may incorrectly assume that the goal is on the right, and become over-confident in this belief 
On the other hand, if the robot only infers the preferred path, it may mistakenly assume that the goal is on the left, leading to a failure in completing the task  To overcome these challenges, our work proposes a joint inference approach that considers both the human's intended goal and their preferred path to that goal 
Specifically, we model the human's preference over different homotopy classes and leverage a conditional independence assumption to provide a tractable solution  In our approach, we assume that the human's inputs are noisily rational conditioned on both the goal and the preference  By jointly inferring the goal and path preference, we can avoid over-confidence in incorrect beliefs about the user's preferences, leading to improved system performance 
We consider the problem of robot navigation in a known environment to an unknown destination, where a human can intervene and provide a heading direction to the robot using a joystick or force cues  The human also has a preference on which path the robot should take with respect to obstacles, and our objective is for the robot to understand the human's intentions and execute the task with minimal interventions 
Let g be a discrete random variable denoting the goal of the task, belonging to a set of candidates Ω g , and let θ be a discrete-valued random variable representing the human's path preference, belonging to a set of possible preferences Θ  The physical location of the robot at time index t is denoted by s t ∈ R 2 , and the robot's action at time index t, belonging to some action space A, is denoted by a t  
The transition model T (s t+1 | s t , a t ) is deterministic, meaning the robot has full control over its future location  At any time step, the human may provide an observation to the robot  When the human intervenes, the robot receives a direction (heading angle) that can be mapped to a future location in space 
More specifically, we map the direction to an intended location, which is the resulting robot location after advancing in the indicated direction for one time step  For simplicity, we consider that the robot directly makes an observation o t of the location indicated by the human  We assume that the robot has a stochastic observation model for the human P (o t | s t , g, θ) that is conditioned on both the goal of the task g and the human's preferred path θ 
We further assume that having chosen a goal and path preference, the human takes actions to noisily minimize a cost function C g,θ that measures the cost of moving from the robot's current location to the goal along the preferred path  For example, C g,θ (s t , o t ) can be the length of the shortest path from location s t to the goal g after taking a first step to o t , and constrained by path preference θ 
We use C g,θ to induce a probability distribution over observations, given by: where γ h is a hyperparameter that designates the rationality coefficient  This model assumes the human will pick the lowest cost action with the highest probability and the likelihood of an action decreases exponentially with the increase in cost  
Our inclusion of the path preference θ sets our approach apart from   The model is shown in fig  represented as a Bayesian Network 

Inference

At each time step where the human provides an observation, the posterior P (g, θ) is given through the Bayesian update We note that the number of Bayesian updates required at each iteration to update the belief is equal to the cardinality of Ω g × Θ  In addition, each Bayesian update involves computing C g,θ (  
,   ) in eq  ( ), which involves solving an optimization problem (such as a shortest path problem)  In section IV, we propose a specific encoding of preference θ for resolving eq  ( ), while ensuring the number of computations of the cost C g,θ ( ,  ) per update does not grow exponentially with the number of obstacles 

Decision Making

We consider a navigation problem where the robot receives reward according to the model R(s t , g, θ, a t )  We wish to find the optimal policy π that maximizes the expected discounted sum of future rewards, with discount factor γ  The above problem is a Partially Observable Markov Decision Process (POMDP)  
In this section, we propose an encoding of human's path preference θ for computing the posterior in eq  ( )  Devifrom the concept of homotopy classes, we define the preference according to a partitioning of the environment into polytopes, as shown in fig  , creating a hyperplane arrangement of the space 
Hyperplane arrangements have been used by Vincent and Schwager in the context of Neural Network verification  In our setting, we leverage this representation to define path preferences as preferred transitions between adjacent regions of the space 

Hyperplane Arrangement

We assume a two-dimensional environment composed of m polytopic obstacles, each defined by their half-space representation (H-representation) where A i ∈ R di×2 and b i ∈ R di , and where d i is the number of edges (hyperplanes) composing polytope i  Let n = i d i be the total number of hyperplanes  We leverage each obstacle's H-representation to construct a hyperplane arrangement of the environment as shown in fig 
 e  a partitioning of the space into polytopes  More specifically, each location in space belongs to a polytope j for which we can write an H-representation of the form where α j i ∈ {−1, 1} di is a vector specific to polytope j and obstacle i corresponding to the relative position of any point in the set with respect to each hyperplane in O i  
Fig  : Intent inference model in a hyperplane arrangement of the obstacle free space  We spatially decompose the preference θ into a set of preferred neighboring polytopes per region of the space  Within each polytope j, the human preference pj is a discrete distribution over the preferred neighbor in N (j) 
We assume that for a location st belonging to polytope j, and given goal g and preference pj, the observation ot and any other preference p i,i =j are conditionally independent  Concatenating elements from each obstacle's Hrepresentation, we can write polytope j's H-representation as where Some of the constraints in eq  ( ) (corresponding to rows of A, b and α j ) are redundant, i e  the set P j does not change upon their removal 
We can further reduce the Hrepresentation of a polytope to include only non-redundant constraints  By removing the rows corresponding to redundant constraints, we obtain new matrices A j e , b j e and α j e such that we can write the polytope's reduced H-representation as The non-redundant constraints correspond to edges of the polytope 
In other words, as the robot continually moves in space, the first hyperplane that it will cross upon exiting the polytope will correspond to one of the polytope's nonredundant constraints  Vincent and Schwager outline an iterative method for removing redundant constraints by solving n linear programs 
We use this method in practice for computing α j e for each polytope  We can now characterize each polytope by a vector α j e ∈ {−1, 1} n j e , where n j e ≤ n is the number of essential constraints of the polytope  The polytopes P j partition the environment into a hyperplane arrangement 

Path Preference

In this section, we provide a definition of preference θ according to a graphical representation of the environment based on the hyperplane arrangement  Under this representation, a path preference corresponds to a set of preferred transitions  In other words, for each polytope in the space, the human will have a preference to which neighboring polytope they wish to transition 
Let G := (V, E) be an undirected graph, where vertices are obstacle-free polytopes, and edges connect two adjacent polytopes  Each polytope is described by a unique vector α j as defined in eq  ( )  Two polytopes are adjacent if they share non-redundant constraints (rows in eq  ( )) corresponding to the same hyperplane (i e  they are on opposite sides of the hyperplane)



ground_truth: SNNs have the potential to better model and explain the functional hierarchy and mechanisms of the visual system.

prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: What is the rationality coefficient used in the observation model?

Read the following text and answer briefly.

Context 3: 
Let N (v) be the set of neighbors of a vertex v  For each vertex, we denote p v the discrete-valued random variable describing which edge in N (v) the human intends to transition to  Using this formalism, we define a path preference as the set of preferred transitions over all nodes in the graph, Let m θ = v∈V |N (v)| be the cardinality of Θ, and m g = |Ω g | the number of possible goals 
A priori, the number of Bayesian updates required to update the belief at every iteration should be m θ × m g   Now, let us assume the conditional independence relationships described by the new problem diagram in fig    More specifically, we introduce the assumption that conditioned on a robot location s t , the goal g, and the preference for the corresponding vertex p v in the graph, the observation o t and the preference for any other vertex are conditionally independent 
In other words, the observations the human provides can be defined conditioned only on the robot location, the goal, and the human's preference for its current vertex p v   By introducing this assumption, each update step only requires updating the joint (p v , g), reducing the number of cost computations to |N (v)| × m g  
We can notice that by introducing this assumption, we removed the direct relationship between the number of polytopes in the environment and the complexity of the Bayesian update in eq  ( )  In practice, components of θ are not mutually independent  For example, if the human preference at a vertex v 1 is
, it is unlikely that the human will also prefer p v2 = (v 2 , v 1 ) (turning back)  We can improve our model by assuming a dependent relationship between preferences for adjacent edges, which does not significantly increase the complexity of the inference problem  An interesting property of our encoding is that any two paths that belong to different homotopy classes will cross different sequences of polytopes, i e  they correspond to a different sequence of edges on G 
This can be proved by contradiction  Let us suppose that two continuous trajectories ξ 1 and ξ 2 , with the same start and end points and that do not intersect any obstacle, traverse the same regions in G in the same order  From the construction of the hyperplane arrangement, each polytope that the paths traverse through is obstacle-free 
Therefore, within each polytope, there is no obstacle in the area located in between the portions of ξ 1 and ξ 2 that belong to the region  A smooth transformation of ξ 1 into ξ 2 can be obtained by transforming each portion of ξ 1 belonging to the polytopes it intersects into the corresponding portion of ξ 2 for the same polytopes, where the extremities of the trajectory portions are connected to one another along the polytope's edges (where the same edge is crossed by both paths) 
Along this transformation, the paths do not intersect any obstacle, and therefore ξ 1 and ξ 2 belong to the same homotopy class 

EXPERIMENTS

We evaluate our model on a simulated navigation task where the robot must reach a goal that is unknown a priori while respecting the path preferences indicated by a human  The robot navigates in a grid world containing obstacles  The transition model is deterministic: the robot selects an adjacent location on the grid to reach at the next time step 
The robot is also allowed to take diagonal actions  Each location s t in the map can be mapped to a vertex v t ∈ G  Therefore, the actions leading to locations mapped to different vertices correspond to edges on the graph  We note f (s t , a t ) the edge crossed by taking action a t from location s t  
The robot is given a mission time limit T max for reaching the goal  In this problem, we assume that the human selects actions to noisily minimize a cost function C g,θ , where θ is defined as per eq  ( ), corresponding to the length of the shortest path to the goal constrained by the preference (where the robot is only allowed to make transitions on G along preferred edges) 
More specifically, where δ(s t , g | o t , p vt ) designates the length of the shortest path from s t to g passing by o t and constrained by preference p vt   This is a slight variant of the cost function proposed by Best and Fitch , where we add in a conditioning on the path preference  We compute costs by running the A path planning algorithm on the environment maps (grid worlds with diagonal actions) and impose preference constraints by pruning invalid transitions from the search tree 
Reward model  At each step in time, the robot receives a reward which is a sum of three components: a goal-specific reward a preference-specific reward or penalty We compute solutions to the POMDP defined in section III-B with the online solver POMCP , and with the particularity that within the rollouts, the robot does not expect to collect human inputs 
Each time a solution is computed, the robot takes an action and may receive an observation  If it does, it updates its belief distribution over the unknown problem variables and resolves the POMDP over a receding horizon 

Baselines

• Goal only  The robot solves the POMDP while ignoring the effects of path preference  Similarly to , we assume the human is taking action to minimize a goaldependent cost C g (s t , o t ) = δ(s t , g | o t ), where the conditioning on the preference is removed  We also omit the path preference's contribution to the reward R pref  
• Compliant  The robot complies with the human input, but does not take an initiative  If the user stops providing information, the robot continues in the last direction indicated for 5 time steps (conserving its momentum), then stops  • Blended  We designed an arbitration function to decide between our proposed policy (accounting for path preferences) and the user's recommendation when the robot receives inputs 
Our metric to evaluate confidence in the robot's prediction for the purpose of arbitration is the entropy of the intention distribution H(g, p i ), where p i denotes the preferred neighbor for the current region  Because our representation of the world is discrete, the arbitration is given by a step function 
Denoted by U , the action corresponding to the human's input, and P , the robot's prediction for the optimal action, we write the policy where we chose h = 1 6 as the confidence threshold 

Results

When evaluating the algorithm, we consider that a run is successful if the robot reached the goal within its allocated mission time T max and only made transitions between graph vertices corresponding to the human's preferences  We vary the time delay between human inputs, from constant guidance (∆ T = 1) to only a single observation (∆ T ≥ T max ) 
Success rates  Table I reports the success rates for experiments conducted over six randomly sampled problem instances and 50 runs per instance in Map 1 (fig  )  When the human provides inputs at every iteration, the compliant policy shows the highest success rates  However, as ∆ T increases, the compliant robot is not able to accomplish the task within the allotted time as it does not receive sufficient inputs to do so, and performance decreases compared to the autonomous baselines 
We find that in these runs, accounting for path preference consistently improves performance compared with the goal-only baseline  Results also show that blending the user's input with the robot's policy (Path Preference + Blend) when the human provides information leads to improved performance  Belief entropy 
Figure shows a challenging problem instance where the directions the human provides do not align directly with the shortest path to the goal  By ignoring the effects of preferences in the problem model (goal only), the robot quickly infers from observations that the upper left goal is less likely than others (P (g) drops) 
The strong decrease in entropy shows that the robot becomes overconfident in this prediction  Overconfidence in an incorrect goal will prevent the agent from finding the correct goal once the human's indications directly align with it, as it needs to correct for the wrong predictions, as shown in the path realization (fig 
)  In this realization, the goal-only method (green robot) fails to search the upper left area within the allotted time  By accounting for path preferences in its model, the blue robot's entropy over the goal distribution decreases more steadily, allowing for it to leverage the human's latest observations and reach the goal successfully 
shows an over-confident prediction (shown by the strong reduction in belief entropy) that the correct goal is less likely, making it more difficult to reach the correct goal compared to a method that accounts for path preference  Computation time  In table II we provide the time required to solve the POMDP, and the time required to update the robot's belief as it receives new observations 
We compute solutions on three maps: a simple 10 × 10 grid world with 8 polytopes (fig  ), a 10 × 10 grid world with 56 polytopes (fig  ), and a 20×20 grid world with 73 polytopes (fig  )  The latter environment being larger, we increase the mission time and the depth of the search tree in POMCP from T max = 30 (Map 1 and Map 2) to T max = 60 (Map 3) 
We do not notice an increase in the time required to update the robot's belief with an increase in problem complexity, which is consistent with our observation that the complexity of the Bayesian update should not increase with the number of obstacles or polytopes  On the contrary, the belief update time on Map 2 and Map 3, containing more obstacles, is reduced compared to the first map 
More obstacles result in fewer iterations when solving the constrained shortest path problem with A   Adding constraints due to the obstacles and polytopes reduces the size of the A search tree  C  Limitations Simulation environments  In our simulations, we hardcoded the preference policy over the maps (e g  in Map 1, go around the table counter-clockwise) 
We randomly sampled problem instances (start and goal locations, and goal options) to reduce the bias introduced by these preference choices  To best evaluate and compare the different approaches, it would be best to sample preferences among a distribution of preferences chosen by a human (for example, from benchmarks resulting from a collection of data) 
Creating such a benchmark is an interesting direction for future work  Hyperplane arrangement construction  The main limitation of our approach is that the size and geometry of each polytope depends strongly on the geometry of the obstacles, as seen in fig    Because of this, the robot can make predictions over preferences that are too refined compared with the topology of the environment 
A direct consequence is that when the size of the polytopes is small, the information provided by the human can be incorrectly interpreted as a preference on the robot's immediate action  Our method can be improved by changing the structure of the hyperplane arrangement so that it relies on the topology of the environment, but does not vary strongly with the geometry of the features in the environment 
For this purpose, topometric maps and region construction algorithms are promising directions  We presented an approach for encoding and inferring a human's path preference in an environment with obstacles  By leveraging a partitioning of the space into polytopes and a stochastic observation model, our method allows for joint inference over the goal and path preference even when both are unknown a-priori 
Our experiments on an unknown-goal navigation problem with sparse human interventions demonstrate the effectiveness of our approach and its suitability for online applications  The time required to update the robot's belief does not increase with the complexity of the environment, which further highlights the practicality of our method 



ground_truth: γh.