prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: What is the potential of SNNs in modeling the visual system?

Read the following text and answer briefly.

Context 1: Paper Info

Title: Deep Spiking Neural Networks with High Representation Similarity Model Visual Pathways of Macaque and Mouse
Publish Date: 22 May 2023
Author List: Zhengyu Ma (from Department of Networked Intelligence, Peng Cheng Laboratory), Yu Liutao (from Department of Networked Intelligence, Peng Cheng Laboratory), Huihui Zhou (from Department of Networked Intelligence, Peng Cheng Laboratory), Allen Brain
Author Affiliation: CORNet-S ConvNeXt-Tiny ConvNeXt-Small EfficientNet, AlexNet RegNetY, ResNet34 ConvNeXt-Base CORNetSEW, ResNet8 ResNet101 SEW-ResNet18 ViT-L, GoogLeNet SEW-ResNet34 SEW-ResNet8 Wide

Figure

Figure 1: To conduct neural representation similarity experiments, we apply three similarity metrics to a layer-by-layer comparison between the responses of models and the neural activities of visual cortex 
Figure 2: For three datasets and three similarity metrics, each point indicates the final representation similarity score of a model Each pair of SEW ResNet and ResNet with the same depth are linked by a gray solid line In almost all conditions, SEW ResNet outperforms ResNet by a large margin 
Figure3: For three datasets and three similarity metrics, we plot the trajectories of similarity score with model layer depth The models are divided into two groups: ResNet and SEW ResNet The normalized layer depth ranges from 0 (the first layer) to 1 (the last layer) Because the depths of models are not the same, we first discretize the normalized depth into 50 bins, and then apply the cubic spline interpolation to the scores of each model, yielding the smooth trajectories shown in the plot The fine, semitransparent lines are the trajectories of each model The thick lines are the average trajectories among each group 
Figure 5: For Macaque-Synthetic dataset, trajectories of similarity score with model layer depth are plotted The models are divided into two groups: ViT and CNN&SNN The normalized layer depth ranges from 0 (the first layer) to 1 (the last layer) The calculation and plotting of the trajectories are the same as Figure 3 
Figure6: The basic block of SpikingMobileNet "PW CONV" is the pointwise convolution and "DW CONV" is the depthwise convolution "SN" is the spiking neuron 
Figure 7: Overall model rankings of the similarity scores on Allen Brain mouse dataset The similarity scores of CNNs, SNNs and vision transformers are shown by blue, green and orange bars, respectively 
Figure 9: Overall model rankings of the similarity scores on Macaque-Synthetic dataset 
Figure 10: The Spearman's rank correlation between the overall model rankings of different metrics There is a strong correlation between SVCCA and TSVD-Reg, but RSA has weaker correlations with them 
The correlation between the similarity scores and the model depth r is Spearman's rank correlation coefficient "-" indicates that there is no significant correlation 
Architectures of SNNs "sn" denotes the spiking neuron "g = 32" denotes the grouped convolutions with 32 groups The hyper-parameters of the spike-element-wise block are shown in the brackets with the number of stacked blocks outside 

abstract

Deep artificial neural networks (ANNs) play a major role in modeling the visual pathways of primate and rodent  However, they highly simplify the computational properties of neurons compared to their biological counterparts  Instead, Spiking Neural Networks (SNNs) are more biologically plausible models since spiking neurons encode information with time sequences of spikes, just like biological neurons do 
However, there is a lack of studies on visual pathways with deep SNNs models  In this study, we model the visual cortex with deep SNNs for the first time, and also with a wide range of state-of-the-art deep CNNs and ViTs for comparison  Using three similarity metrics, we conduct neural representation similarity experiments on three neural datasets collected from two species under three types of stimuli 
Based on extensive similarity analyses, we further investigate the functional hierarchy and mechanisms across species  Almost all similarity scores of SNNs are higher than their counterparts of CNNs with an average of 6 6%  Depths of the layers with the highest similarity scores exhibit little differences across mouse cortical regions, but vary significantly across macaque regions, suggesting that the visual processing structure of mice is more regionally homogeneous than that of macaques 
Besides, the multi-branch structures observed in some top mouse brain-like neural networks provide computational evidence of parallel processing streams in mice, and the different performance in fitting macaque neural representations under different stimuli exhibits the functional specialization of information processing in macaques 
Taken together, our study demonstrates that SNNs could serve as promising candidates to better model and explain the functional hierarchy and mechanisms of the visual system  Originally, the prototype of deep neural networks is inspired by the biological vision system   To date, deep neural networks not only occupy an unassailable position in the field of computer vision , but also become better models of the biological visual cortex compared to traditional models in the neuroscience community (Khaligh-Razavi and Kriegeskorte 2014;  
They have been successful at predicting the neural responses in primate visual cortex, matching the hierarchy of ventral visual stream (Güc ¸lü and van Gerven 2015; , and even controlling neural activity   Moreover, as training paradigms of mice and techniques for collecting neural activity (de Vries et al  2020) have been greatly improved, there is a strong interest in exploring mouse visual cortex 
Deep neural networks also play an important role in revealing the functional mechanisms and structures of mouse visual cortex   Compared to biological networks, Artificial Neural Networks discard the complexity of neurons   Spiking Neural Networks, incorporating the concept of time and spikes, are more biologically plausible models  
To be more specific, because of their capabilities of encoding information with spikes, capturing the dynamics of biological neurons, and extracting spatio-temporal features, deep SNNs are highly possible to yield brain-like representations )  However, deep SNNs have not been employed to model visual cortex due to the immaturity of training algorithms 
Recently, a state-ofthe-art directly trained deep SNN , makes it possible to use deep SNNs as visual cortex models  Contributions  In this work, we conduct large-scale neural representation similarity experiments on SNNs and other high-performing deep neural networks to study the brain's visual processing mechanisms, with three datasets and three similarity metrics (Figure ) 
Specifically, to the best of our knowledge, we are the first to use deep SNNs to fit complex biological neural representations and explore the biological visual cortex  We summarize our main contributions in four points as follows  • We find that SNNs outperform their counterparts of CNNs with the same depth and almost the same architectures in almost all experiments 
In addition, even with very different depths and architectures, SNNs can achieve top performance in most conditions  • By making a more direct comparison between macaques and mice for the first time, we reveal the differences in the visual pathways across the two species in terms of the homogeneity of visual regions and the increases of receptive field sizes across cortical visual pathways, which is consistent with previous physiological work 
• The multi-branch structures in neural networks benefit neural representation similarity to mouse visual cortex, providing computational evidence that parallel information processing streams are widespread between cortical regions in the mouse visual system  • Comparing the results of two macaque neural datasets under different stimuli, we reveal that the macaque vision system may have functional specialization for processing human faces and other natural scenes 
Altogether, as the first work to apply deep SNNs to fit neural representations, we shed light on visual processing mechanisms in both macaques and mice, demonstrating the potential of SNNs as a novel and powerful tool for research on the visual system  Our codes and appendix are available at https://github com/Grasshlw/SNN-Neural-Similarity 
There are plenty of computational models of macaque and mouse visual systems for exploring the visual processing mechanisms recently  We summarize some of the outstanding work in the following  The network models of macaque visual system  In the early days, studies basically used simple feedforward neural networks as the models of the macaque visual system (Khaligh-Razavi and Kriegeskorte 2014;  
Recently, some bio-inspired or more complex models achieved better performance in fitting the neural representations of macaque visual cortex   proposed a brainlike shallow CNN with recurrent connections to better match the macaque ventral visual stream  By mimicking the primary stage of the primate visual system, VOneNets ) performed more robustly in image recognition while better simulating macaque V1 
Moreover, the representations learned by unsupervised neural networks ) also effectively matched the neural activity of macaque ventral visual stream  Although the above work developed many bio-inspired structures, the networks are still traditional ANNs in nature  Our work introduces deep SNNs for the first time to explore the visual processing mechanisms of macaque visual system 
The network models of mouse visual system  Largescale mouse neural dataset provided an experimental basis for model studies of mouse visual system (de Vries et al  2020;   conducted comparisons between the representations of mouse visual cortex and the VGG16 trained on the Im-ageNet dataset  In , they developed a single neural network to model both the dorsal and ventral pathways with showing the functional specializations 
What's more, a large survey of advanced deep networks ) revealed some hierarchy and functional properties of mice  Similar to the studies of macaque visual system, deep SNNs have never been used to model the mouse visual system  In this work, we not only use SNNs as one of the candidates to fit the representations of mouse visual cortex, but also conduct direct comparisons between macaques and mice to further investigate the functional hierarchy and mechanisms of the two species 
Our work is conducted with three neural datasets  These datasets are recorded from two species under three types of stimuli  More specifically, there are neural responses of mouse visual cortex to natural scene stimuli, and responses of macaque visual cortex to face image and synthetic image stimuli  Allen Brain mouse dataset 
It is part of the Allen Brain Observatory Visual Coding dataset ) col-lected using Neuropixel probes from 6 regions simultaneously in mouse visual cortex  Compared to two-photon calcium imaging, Neuropixel probes simultaneously record the spikes across many cortical regions with high temporal resolution 
In these experiments, mice are presented with 118 250-ms natural scene stimuli in random orders for 50 times  Hundreds to thousands of neurons are recorded for each brain region  To get the stable neurons, we first concatenate the neural responses (average number of spikes in 10-ms bins across time) under 118 images for each neuron, and then preserve the neurons whose split-half reliability across 50 trials reaches at least 0 8 
Macaque-Face dataset  This dataset ) is composed of neural responses of 159 neurons in the macaque anterior medial (AM) face patch under 2,100 real face stimuli, recorded with Tungsten electrodes  For this dataset, we compute the average number of spikes in a time window of 50-350ms after stimulus onset and exclude eleven neurons with noisy responses by assessing the neurons' noise ceiling 
The details of the preprocessing procedure are the same as   Macaque-Synthetic dataset  This dataset is also about macaque neural responses which are recorded by electrodes under 3,200 synthetic image stimuli, and used for neural prediction in the initial version of Brain-Score   The image stimuli are generated by adding a 2D projection of a 3D object model to a natural background 
The objects consist of eight categories, each with eight subclasses  The position, pose, and size of each object are randomly selected  88 neurons of V4 and 168 neurons of IT are recorded  The neural responses are preprocessed to the form of average firing rate and can be downloaded from Brain-Score  Since the core visual function of macaque and mouse visual cortex is to recognize objects, the basic premise of model selection is that the model has good performance on object recognition tasks (e g 
classification on ImageNet)  Based on this premise, we employ 12 SNNs, 43 CNNs, and 26 vision transformers, all of which are pretrained on the Ima-geNet dataset and perform well in the classification task  As for SNNs, we use SEW ResNet as the base model, which is the deepest and SOTA directly trained SNN  
Furthermore, by combining the residual block used in SEW ResNet and the hierarchy of the visual cortex, we build several new SNNs and train them on the ImageNet using SpikingJelly ) (see Appendix A for model structures and the details of model training)  As for CNNs and vision transformers, we use 44 models from the Torchvision model zoo , 22 models from the Timm model zoo ) and 3 models from the brain-like CNNs, CORnet family ) 
In the feature extraction procedures of all models, we feed the same set of images used in biological experiments to the pretrained models and obtain features from all chosen layers  Different from CNNs and vision transformers, the features of SNNs are spikes in multiple time steps  To obtain the representation similarity between biological visual cortex and computational models, we apply three similarity metrics to computing similarity scores: representational similarity analysis (RSA) , regression-based encoding method and singular vector canonical correlation analysis (SVCCA)  
RSA has already been widely used to analyze neural representations of a model and a brain to different stimuli at the population level, while the regression-based encoding method directly fits the model features to neural activity data  SVCCA is originally proposed to compare features of deep neural networks, and then Buice 2019) used it to compare representation matrices from mouse visual cortex and DNNs, which demonstrated its effectiveness 
With the same model and same cortical region, we use these metrics for a layer-by-layer comparison to compute the similarity scores  The maximum similarity score across layers for a given cortical region is considered to be the level of representation similarity between the model and the cortical region 
Finally, in a given dataset, we take the average score of all cortical regions as the final similarity score for each model, which gives the overall model rankings  The implementation of each similarity metric is as follows  RSA  For two response matrices R ∈ R n×m from each layer of models and each cortical region, where n is the number of units/neurons and m is the number of stimuli, we calculate the representational similarity between the responses to each pair of image stimuli using the Pearson correlation coefficient r, yielding two representational dissimilarity matrices (RDM ∈ R m×m , where each element is the correlation distance 1 − r) 
Then, the Spearman rank correlation coefficient between the flattened upper triangles of these two matrices is the metric score  Regression-Based Encoding Method  Firstly, we run truncated singular value decomposition (TSVD) to reduce the feature dimension of model layers to 40  Secondly, the features after dimensionality reduction are fitted to the representations of each neuron by ridge regression 
Finally, we compute the Pearson correlation coefficient between the predicted and ground-truth representations of each neuron and take the mean of all correlation coefficients as the metric score  More specifically, we apply leave-one-out crossvalidation to obtain predicted representations of each neuron 
For simplicity, we name this method 'TSVD-Reg'  SVCCA  For both the responses of model layers and cortical regions, we use TSVD to reduce the dimension of unit/neuron to 40, yielding two reduced representation matrices  Then we apply canonical correlation analysis (CCA) to these two matrices to obtain a vector of correlation coefficients (the length of the vector is 40) 
The metric score is the mean of the vector  Because of the invariance of CCA to affine transformations , in this procedure, we only need to ensure that the stimulus dimension is consistent and aligned, even if the unit/neuron dimension is different  Dimensionality reduction plays an important role in this method to make the number of model features comparable to the number of neurons in cortical regions, since the former usually far exceeds the latter 
In addition, dimensionality reduction helps to determine which features are important to the original data, while CCA suffers in important feature detection  Using just CCA performs badly, which has been proven by   To check how similar the models are to the visual cortex's mechanisms in visual processing, we rank the final similarity scores of all models and conduct comparisons among three types of models (CNNs, SNNs, and vision transformers) 
Specially, we focus on comparing SNN (SEW ResNet) and CNN (ResNet) with the same depth and almost the same architectures (Figure )  The final similarity score of a model is the average similarity score across all cortical regions  (The overall rankings can be found in Appendix B and the comparisons among three types of models are shown in Appendix C )
Allen brain mouse dataset  No single model achieves the highest final similarity scores with all three metrics  For a fair comparison, we apply the paired t-test to SEW ResNet and ResNet with the same depth  For all three metrics, SEW ResNet performs better than ResNet by a large margin (t = 5 857, p = 0 004; t = 7 666, p = 0 002; t = 7 592, p = 0 002) 1   1 The results of the three similarity metrics are separated by semicolons, in the order of SVCCA, TSVD-Reg, and RSA 
Other Macaque-Face dataset  For both SVCCA and TSVD-Reg, Wide-SEW-ResNet14 and Wide-SEW-ResNet8 achieve the first and second highest final similarity scores respectively  But for RSA, TNT-S and Inception-ResNet-V2 take their place and outperform other models by a large margin  As for SEW ResNet and ResNet, the former performs significantly better than the latter for both SVCCA and TSVD-Reg (t = 8 195, p = 0 001; t = 7 528, p = 0 002) 
However, the difference is not significant for RSA (t = 1 117, p = 0 327)  Specifically, the similarity score of SEW ResNet152 is only slightly higher than that of ResNet152, and at the depth of 50 and 101, SEW ResNet's scores are lower than ResNet's  Macaque-Synthetic dataset  Similar to the results of Allen Brain dataset, no model performs best for all three metrics 
SEW ResNet performs moderately better than ResNet (t = 3 354, p = 0 028; t = 3 824, p = 0 019; t = 2 343, p = 0 079)  The only contrary is that SEW ResNet18 performs worse than ResNet18 for RSA  Further, to check the details of comparison between the SNNs and their CNN counterparts, we analyze the trajectories of similarity score across model layers (Figure ) 
As for ResNet and SEW ResNet with the same depth, the trends of their similarities across model layers are almost the same, but the former's trajectory is generally below the latter's  In other words, the similarity scores of SEW ResNet are higher than those of ResNet at almost all layers  Taken together, the results suggest that when the overall results that appear below also correspond to the three metrics in this order, unless the correspondence is stated in the text 
architectures and depth are the same, SNNs with spiking neurons perform consistently better than their counterparts of CNNs with an average increase of 6 6%  Besides, SEW ResNet14 also outperforms the brain-like recurrent CNN, CORnet-S, with the same number of layers (see more details in Appendix B)  Two properties of SNNs might contribute to the higher similarity scores 
On the one hand, IF neurons are the basic neurons of spiking neural networks  The IF neuron uses several differential equations to roughly approximate the membrane potential dynamics of biological neurons, which provides a more biologically plausible spike mechanism for the network  On the other hand, the spiking neural network is able to capture the temporal features by incorporating both time and binary signals, just like the biological visual system during information processing 
To figure out the distinctions in the functional hierarchy between macaques and mice, for each cortical region, we obtain the normalized depth of the layer that achieves the highest similarity score in each model  Then, we divide models (excluding vision transformers) into two groups based on their depths and conduct investigations on these two groups separately 
A nonparametric ANOVA is applied to each group for testing whether layer depths change significantly across cortical regions  For mouse visual cortex (Figure (a)), taking the deep model group as an example, ANOVA shows overall significant changes in depth across cortical regions for TSVD-Reg and RSA (Friedman's χ 2 = 49 169,
p = 2 0 × 10 −9 ; χ 2 = 19 455, p = 0 002)  But there is no significant change for SVCCA (χ 2 = 8 689, p = 0 122)  According to these results, the differences in depth across regions are indeterminacy and irregular  Meanwhile, the trends of layer depth between some regions contradict the hierarchy observed in physiological experiments of mice (those between VISp and VISrl for TSVD-Reg and between VISal and VISpm for RSA) 
However, for macaque visual cortex (Figure (b)), there are significant differences (t = −5 451, p = 6 5 × 10 −6 ; t = −8 312, p = 2 8 × 10 −9 ; t = −3 782, p = 6 9 × 10 −4 , also taking the deep model group as an example) between V4 and IT, and the trend is consistent with the information processing hierarchy in primate visual cortex 
The comparative analyses of the best layer depths of the shallow and deep model groups also exhibit the differences between macaques and mice  For mouse visual cortex, the best layer depths of shallow models are significantly higher than those of deep models  Compared to deep models, most shallow models achieve the top similarity scores in intermediate and even later layers 
Differently, for macaque visual cortex, the depth of models has little effect on the depth of the most similar layer  What's more, we find that the most similar layer of mouse visual cortex always occurs after the 28 × 28 feature map is downsampled to 14 × 14, which leads to the layer depths' difference between shallow and deep models 
Nevertheless, the best layer of macaque IT appears in the last part of networks, where the feature map has been downsampled more times  In summary, our results might reveal two distinctions in the functional hierarchy between macaques and mice  First, there is a distinct functional hierarchical structure of macaque ventral visual pathway, while there might be no clear sequential functional hierarchy in mouse visual cortex 
One explanation is that the mouse visual cortex is organized into a parallel structure and the function of mouse cortical regions are more generalized and homogeneous than those of macaques  Another possibility would be that even though the sequential relations exist among mouse cortical regions as proposed in anatomical and physiological work, they are too weak for the current deep neural networks to capture 
Additionally, mice perform more complex visual tasks than expected with a limited brain capacity   Consequently, the neural responses of mouse visual cortex may contain more information not related to object recognition that neural networks focus on  Secondly, it is well known that the units in the neural networks get larger receptive fields after downsampling, and through the analyses of differences between two groups of models based on depth, we find the feature map of the best layer for mouse is downsampled fewer times than that for macaque 
Based on these results, we provide computational evidence that the increased ratio of the receptive field size in cortical regions across the mouse visual pathway is smaller than those across the macaque visual pathways, which echoes some physio- Macaque-Face dataset --- Table : The correlation between the similarity scores and the number of parameters 
r is Spearman's rank correlation coefficient  "-" indicates that there is no significant correlation  To explore the processing mechanisms in the visual cortex of macaques and mice, we investigate the model properties from the whole to the details  As shown in Table and 2, we first measure the correlation between the similarity scores and the sizes (i e  the number of trainable parameters and the depth) of network models 
For Allen Brain mouse dataset, there are significant negative correlations between the similarity scores and the number of parameters for three metrics while there is no correlation with the depth  Conversely, for the two macaque neural datasets, the similarity scores are highly correlated with the depth of networks, but not with the number of parameters 
Specifically, there is a positive correlation for Macaque-Face dataset while a negative correlation for Macaque-Synthetic dataset  (We also apply the linear regression to analyze the correlation between the similarity scores and the model size  The results are consistent with Spearman's rank correlation and are shown in Appendix E) 
Based on these results, we further investigate more detailed properties of neural networks to explain the processing mechanisms in the visual cortex  For the mouse dataset, on the one hand, the best layer depths show non-significant changes across the mouse cortical regions as mentioned in the previous section 
On the other hand, the similarity scores of the mouse dataset are only correlated with the number of model parameters but not with the depth of models  It calls into the question whether any detailed structures in the neural networks help to reduce the number of parameters and improve its similarity to mouse visual cortex 
Therefore, we explore the commonalities between models that have the top 20% representation similarities (see Appendix D) for Allen Brain dataset  As expected, the top models contain similar structures, such as fire module, inception module, and depthwise separable convolution  All these structures essentially process information through multiple branches/channels and then integrate the features from each branch 
The models with this type of structure outperform other models (t = 2 411, p = 0 024; t = 3 030, p = 0 007; t = 1 174, p = 0 247)  Moreover, we apply the depthwise separable convolution to SNNs, which yields a positive effect  The representation similarity of Spiking-MobileNet is higher than SEW-ResNet50 with a similar depth (+0 8%; +3 9%; +12 1%) 
In fact, some studies using multiple pathways simulate the functions of mouse visual cortex to some extent   Our results further suggest that not only the mouse visual cortex might be an organization of parallel structures, but also there are extensive parallel information processing streams between each pair of cortical regions  
For the two macaque datasets with different stimuli, not only are the model rankings significantly different, but also the correlations between the similarity scores and the model depth are totally opposite  These results corroborate the following two processing mechanisms in macaques: the ventral visual stream of primate visual cortex possesses canonical coding principles at different stages; the brain exhibits a high degree of functional specialization, such as the visual recognition of faces and other objects, which is reflected in the different neural responses of the corresponding region (although the face patch AM is a sub-network of IT, they differ in the neural representations) 
Besides, as shown in Figure , The calculation and plotting of the trajectories are the same as Figure   the similarity scores of vision transformers reach the maximum in the early layers and then decrease  Differently, the scores of CNNs and SNNs keep trending upwards, reaching the maximum in almost the last layer 
On the other hand, Appendix C shows that vision transformers perform well in Macaque-Face dataset but poorly in Macaque-Synthetic dataset  Considering the features extraction mechanism of vision transformers, it divides the image into several patches and encodes each patch as well as their internal relation by self-attention 
This mechanism is effective for face images that are full of useful information  However, the synthetic image consists of a central target object and a naturalistic background  When vision transformers are fed with this type of stimuli, premature integration of global information can lead to model representations containing noise from the unrelated background 
What's more, when we take all models with the top 20% representation similarities as a whole for analyses, as described in the above paragraph, the properties that enable networks to achieve higher neural similarity are not yet clear  Taken together, the computational mechanism of the better models may reveal core processing divergence to different types of stimuli in the visual cortex 
In this work, we take large-scale neural representation similarity experiments as a basis, aided by analyses of the similarities across models and the visual cortical regions  Compared to other work, we introduce SNNs in the similarity analyses with biological neural responses for the first time, showing that SNNs achieve higher similarity scores than CNNs that have the same depth and almost the same architectures 
As analyzed in Section 3 1, two properties of SNNs might serve as the explanations for their high similarity scores  The subsequent analyses of the models' simulation performance and structures indicate significant differences in functional hierarchies between macaque and mouse visual cortex  As for macaques, we observed a clear sequential hi-erarchy 
However, as for mouse visual cortex, some work ) exhibits that the trend of the model feature complexity roughly matches the processing hierarchy, but other work suggests that the cortex ) is organized into a parallel structure  Our results are more supportive of the latter  Furthermore, we provide computational evidence not only that the increased ratio of the receptive field size in cortical regions across the mouse visual pathway is smaller than those across the macaque visual pathway, but also that there may be multiple pathways with parallel processing streams between mouse cortical regions 
Our results also clearly reveal that the processing mechanisms of macaque visual cortex differ to various stimuli  These findings provide us with new insights into the visual processing mechanisms of macaque and mouse, which are the two species that dominate the research of biological vision systems and differ considerably from each other 
Compared to CNNs, the study of task-driven deep SNNs is just in its initial state  Although we demonstrate that SNNs outperform their counterparts of CNNs, SNNs exhibit similar properties as CNNs in the further analyses  In this work, we only build several new SNNs by taking the hints from the biological visual hierarchy, while many well-established structures and learning algorithms in CNNs have not been applied to SNNs yet 
In addition, the neural datasets used in our experiments are all collected under static image stimuli, lacking rich dynamic information to some certain, which may not fully exploit the properties of SNNs  Given that SNNs perform well in the current experiments, we hope to explore more potential of SNNs in future work 
In conclusion, as more biologically plausible neural networks, SNNs may serve as a shortcut to explore the biological visual cortex  With studies on various aspects of SNNs, such as model architectures, learning algorithms, processing mechanisms, and neural coding methods, it's highly promising to better explain the sophisticated, complex, and diverse vision systems in the future 

Implementation Details of SNNs Spiking Neuron Model

For all SNNs, we use the Integrate-and-Fire (IF) model as the spiking neuron model, which acts as the activation layer in neural networks  As mentioned in , V t , X t and S t denote the state (membrane voltage), input (current) and output (spike) of the spiking neuron model respectively at time-step t, and the dynamics of the IF model can be described as follows:
(1) (2) (3) While V t is the membrane voltage after the trigger of a spike, H t is also the membrane voltage, but after charging and before a spike firing  Θ(x) is the unit step function, so S t equals 1 when H t is greater than or equal to the threshold voltage V thresh and 0 otherwise  Meanwhile, when a spike fires, V t is reset to V reset  
Here, we set V thresh = 1 and V reset = 0  In addition, because Θ(x) is non-differentiable at 0, the surrogate gradient method is applied to approximate the derivative function during back-propagation  Here, we use the inverse tangent function as the surrogate gradient function and the derivative function is
(5) In our experiments on SNNs, we not only use SEW ResNet proposed by ), but also build several new SNNs  On the one hand, we improve the spike-elementwise block in SEW ResNet with new architectures referring to studies on ResNet , as shown in Table   On the other hand, as the multi-branch structures in CNNs increase neural representation similarity to mouse visual cortex, we use depthwise separable convolutions and follow the overall architecture of MobileNetV2 to build the SpikingMobileNet, the basic block of which is shown in Figure  
Our implementation is based on SpikingJelly , an open-source framework of deep SNN  We use the ImageNet dataset to pre-train the new SNNs  Following the settings for training SEW ResNet , we train the models for 320 epochs on 8 GPUs (NVIDIA V100), using SGD with a mini-batch size of 32  The momentum is 0 9 and the weight decay is 0  The initial learning rate is 0 1 and we decay it with a cosine annealing, where the maximum number of iterations is the same as the number of epochs 
For all SNNs, we set the simulation duration T = 4 

Overall model rankings

The results of model rankings are shown in Figure , 8 and 9  We also apply the Spearman's rank correlation to the overall model rankings of different metrics, which is shown in Figure  

Score Comparisons among Model Groups

We conduct comparisons of similarity scores among CNNs, SNNs, and vision transformers  The results are shown in Figure  

Overall CNN rankings

The results of CNN rankings are shown in Figure , 13 and 14 

Correlations between the Model Sizes and the Similarity Scores

The results of linear regression to model sizes and the similarity scores are shown in Figure , 16 and 17 

The ImageNet Accuracy and the Similarity Scores

The results are shown in Figure  
Context 2: \section{Introduction}
Cognitive deficit of older adults is one of the biggest global public health challenges in elderly care  Approximately 5 2 million people of 65 and older are suffered with any form of cognitive impairments in United States in 2012 \cite{stat12}  Dementia is one of the major causes of the cognitive impairments which is more acute among 85 and older population (50\%) \cite{stat12}  However, the costs (financial and time) of health care and long-term care for individuals with Alzheimer's (special form of dementia) or other dementias are substantial  For example, during 2016, about 15 9 million family and friends in United States provided 18 2 billion hours of unpaid assistance to those with cognitive impairments which is a contribution to the nation valued at \$230 1 billion  One the other hand, total payments for all individuals with all form of cognitive impairments are estimated at \$259 billion  Total annual payments for health care, long-term care and hospice care for people with Alzheimer's or other dementias are projected to increase from \$259 billion in 2017 to more than \$1 1 trillion in 2050  Among the above costs, a significant amount are relevant to clinical and diagnostic tests \cite{stat17}  Although clinical and diagnostic tests have become more precise in identifying dementia, studies have shown that there is a high degree of underrecognition especially in early detection  However, there are many advantages to obtaining an early and accurate diagnosis when cognitive symptoms are first noticed as the root cause findings of impairment always lessen the progress of impairment status and sometimes symptoms can be reversible and cured 

With the proliferation of emerging ubiquitous computing technologies, many mobile and wearable devices have been available to capture continuous functional and physiological behavior of older adults  Wearable sensors are now capable of estimating number of steps being taken, physical activity levels, sleep patterns and physiological outcomes (heart rate, skin conductance) of older adults \cite{sano15}  Ambient sensors also help capture the movement patterns of objects and humans for activity and behavior recognition \cite{dawadi14,dawadi15}  Researchers also proved the existence of correlations between cognitive impairment and everyday task performance \cite{dawadi14, akl15,alam16} as well as physiological symptoms \cite{alam16,sano15}  Although current studies showed some successes in IoT-assisted cognitive health assessment in different domains individually, there are several existing challenges in developing and validating a fully automated multi-modal assessment model 

\begin{enumerate}
\item \emph{Real-time IoT System}: A real-time IoT system must include a continuous and fault tolerant data streaming capability among central hub, wearable sensors and ambient sensors regardless of network communication protocol (WiFi, Ethernet, Bluetooth etc ) which are not available in existing researches 
\item \emph{Multi-modal Context Fusion}: Though several offline clinically validated cognitive health assessment tools exist \cite{wai03, starling99, krapp07, yesavage82, zung71}, there is no universally accepted method for IoT-assisted automatic cognitive health assessment in smart home environment that can fuse multi-modal sensor contexts altogether  For example, some researchers showed ambient sensors based Activities of Daily Livigin (ADLs) sequence pattern can signify the cognitive health status of older adults \cite{akl15, dawadi15}  Researchers also showed wearable Electrodermal Activity pattern analysis may carry the significance of cognitive status \cite{sano15}  However, for validation of IoT based cognitive health assessment, self-reported surveys, clinical diagnosis and observation based tools are used individually by prior researchers \cite{akl15, dawadi15, sano15, alam16} 
\end{enumerate}

Regarding aforementioned challenges for the automation of cognitive health assessment, \emph{AutoCogniSys} considers (i) reproducibility of our model in any smart home system consists of ambient motion sensors, wearable accelerometer (ACC) sensors, wearable Electrodermal Activity (EDA) and Photoplethysmography (PPG) sensors individually or combined streams; (ii) context awareness based on ambient motion sensors and wearable ACC sensors in any types of activities such as hand gestural, postural and complex ADLs; and (iii) high accuracy, i e , a recall rate of over 90\% with less than 5\% false positive rate  More specifically, \emph{AutoCogniSys} extends our existing work \cite{alam16} in three dimensions,

\emph{(1) True Automation:} We first investigate the correlations of cognitive impairment with human activities and stress where we manually labeled activities, extract the corresponding physiological sensor (EDA and PPG) features of each activity, and use statistical method to find correlations  Then, we propose automatic complex activity recognition based on a Hierarchical Dynamic Bayesian Network (HDBN) model, fine-grained extraction of physiological sensor features and finally machine learning classification of cognitive impairment 

\emph{(2) Noises Elimination:} We define different types of noises on ACC, EDA and PPG sensors, propose extensive signal processing techniques to remove noises and show significant improvement can be achieved in cognitive impairment classification 

\emph{(3) Implementation and Evaluation:} Finally, we design and implement IoT system and analytic methods and minimize the human involvement to automate our proposed cognitive health assessment approach by considering effective smart home sensor customization and deployment, data collection, screening, cleaning and filtering, feature computation, normalization and classification, and activity  model training 

\textbf{Research Questions:} \emph{AutoCogniSys} consequently tackles the following key research questions 

$\bullet$ Can we detect simultaneously the periodic rhythms of  both hand gestures and postural activities from wrist-worn ACC sensor signal for diverse population (population with same activity but diverse ways such as walking with walker, stretcher or normally)? If so, how can we incorporate the hand gesture, posture and ambient sensor data streams to help improve the ADLs recognition models?

$\bullet$ How can we exploit and relate the micro-activity features into noise free physiological sensor signals processing to automate cognitive health assessment process? What are the critical roles of clinical survey and technology guided assessment methodologies and their inter-relationships for automating the different intermediate steps of cognitive health assessment process?

To tackle these, we make the following \textbf{key contributions}:

$\bullet$ We employ an extensive signal deconvolution technique that in conjunction with machine learning technique helps facilitate a wrist-worn ACC-based multi-label (hand gestural and postural) activity recognition for diverse population  We then leverage multi-label context sets with ambient and object sensor signals for complex activity recognition based on HDBN model 

$\bullet$ We propose a novel collaborative filter for EDA signal processing by postulating signal as a mixture of three components: \emph{tonic phase, phasic phase} and \emph{motion artifacts}, and employ convex optimization technique for filtering out the motion artifacts  We also propose a novel PPG signal processing technique to filter out the inherent motion artifacts and noises using improved Periodic Moving Average Filtering (PMAF) technique 

$\bullet$  We design and prototype an IoT system consisting of multiple devices (wearable wrist band, IP camera, object and ambient sensors) connected with central hub via WiFi, Ethernet and Bluetooth communication protocols  We collected data from 22 older adults living in a continuing care retirement community center in a very natural setting (IRB \#HP-00064387) 

$\bullet$ Finally, we employ statistical and machine learning techniques to jointly correlate the activity performance metrics and stress (EDA and PPG) features that helps achieve max  93\% of cognitive impairment status detection accuracy  We evaluate \emph{AutoCogniSys} on 5 clinically validated offline assessment tools as ground truth 
\section{Related Works}
\emph{AutoCogniSys} builds on previous works on wearable devices based low-level (postural and hand gestural) activity recognition and their integration with ambient sensors to recognize complex ADLs, the underlying signal processing and applications on cognitive health assessment automation 
\subsection{Wearable Sensor Signal Processing}
Wearable sensors can be two types: physical and physiological  Physical sensors (accelerometer, gyroscope etc ) signal values change over the movements of the sensor devices  Physiological sensors change over physiological condition of body such as EDA changes over stress and PPG changes over heart rate  However, physical movements also impose noises on physiological sensor signals which is called \emph{motion artifacts} 
\subsubsection{Physiological Signal Processing}
A continuous and descrete decomposition of EDA, and time and frequency domain analytics of PPG signal have been investigated before to extract relevant physiological features which were contaminated with noises and motion artifacts \cite{alam16}  \cite{setz10} denoised and classified EDA from cognitive load and stress with accuracy higher than 80\%  Though motion artifacts removal techniques such as exponential smoothing \cite{hern11} and low-pass filters \cite{poh10, hernandez14} provide significant improvement in filtering EDA signals, wavelet transforms offer more sophisticated refinement for any kind of physiological sensors such as electroencephalogram \cite{krish06, zikov02}, electrocardiogram \cite{erc06,alfa08}, and PPG \cite{lee03}  \cite{chen15} proposed a stationary wavelet transform (SWT) based motion artifacts removal technique  `cvxEDA' proposed  a convex optimization technique considering EDA as a mixture of white gaussian noise, tonic and phasic components where white gaussian noise includes motion artifacts and external noises \cite{greco16}  \emph{AutoCogniSys} intelligently combines SWT and `cvxEDA' together to remove noises and motion artifacts from EDA signal  On the other hand, it is more difficult to remove motion artifacts from PPG signal due to its periodicity of nature \cite{wang13}  Researchers proposed different methods such as frequency analytics \cite{garde13,wang13}, statistical analytics \cite{peng14} and digital filter \cite{lee10} to reduce noises and motion artifacts from PPG  \emph{AutoCogniSys} used Periodic Moving Average Filter (PMAF) in this regard \cite{lee07} 
\subsubsection{Physical Sensor Signal Processing}
ACC based hand gesture recognition has been explored by several researchers in past such as discrete hidden markov model \cite{liu10}, artificial neural network \cite{arce11}, weighted naive bayes and dynamic time warping \cite{mace13}  Akl et  al  proposed 18 gesture dictionary based Support Vector Machine (SVM) classifier \cite{akl11}  Wrist-worn ACC based postural activity recognition approach has been proposed using Decision Tree, Random Forest, Support Vector Machines, K-Nearest Neighbors, Naive Bayes and deep neural networks \cite{gj14, wang16}, the accuracy stagnates at 85\% using SVM method \cite{martin16}  However, neither of past works proposed any technique that can provide single body worn ACC sensor-based multiple body contexts recognition nor works efficiently for diverse posture say walking normally, with walker, with double walker or wheel chair  Our proposed 8-hand gesture recognition technique assisted sparse-deconvolution method improves classification performances on both normal and diverse postures  However, we incorporated hand gestures and postures in conjunction with ambient sensors into single-inhabitant HDBN model \cite{alam16b} that provides significant improvement in complex activity recognition 
\subsection{Cognitive Health Assessment}
Smart home environment has been used for providing automated health monitoring and assessment in the ageing population before \cite{dawadi14, gong15, akl15, dawadi15}  `SmartFABER' proposed a non-intrusive sensor network based continuous smart home environmental sensor data acquisition and a novel hybrid statistical and knowledge-based technique to analyz the data to estimate behavioral anomalies for early detection of mild-cognitively impairment \cite{riboni16}  \cite{skubic15} presented an example of unobtrusive, continuous monitoring system for the purpose of assessing early health changes to alert caregivers about the potential signs of health hazards  Though, prior researches proposed a sequence of ambient motion sensor streams as complex activity components in activity based health assessment \cite{dawadi14, gong15, akl15, dawadi15}, we consider inclusion of an wearable wrist-band with in-built ACC sensor to detect hand gesture and posture, augmenting with the ambient sensor readings to help recognize complex activities as well as cognitive health assessment of older adults  Additionally, we propose intelligent use of physiological features of skin through different physiological sensor signals (EDA, PPG) processing in daily activity tasks and incorporate context-awareness for automation of cognitive health assessment that have not been explored before 
\begin{figure}[!htb]
\begin{center}
   \epsfig{file=flowchart pdf,height=1 6in, width=3 5in}
\caption{Overall flow of \emph{AutoCogniSys} pipeline }
   \label{fig:overview}
\end{center}
\end{figure}
\section{Overall Architecture}
We first investigate existing IoT-based cognitive health care frameworks that covers every aspects of wearable (physical, physiological) and ambient (passive infrared and object) sensor signals computing  \emph{AutoCogniSys} is comprised of three component modules: (i)~sensing, (ii)~processing, and (iii)~analysis  The `sensing' module consists of clinical assessment tools (surveys, observation and clinical backgrounds) and sensing signals (ambient and wearable sensors)  `Sensor processing' module is comprised of three sub-modules: a)~clinical assessment feature extraction from assessment tools; b)~ambient sensor feature extraction; and c)~wearable sensor processing (noise removal, segmentation, feature extraction, classification etc )  `Analysis' module is comprised of machine learning and statistical analytics-based score prediction of cognitive impairment  Automation of each module's functionality and inter-intra modular transactions without human interference can be called {\it true automation} of cognitive health assessment  Fig ~\ref{fig:overview} shows the overall flow of \emph{AutoCogniSys} which is discussed in details in the following sections 
\subsection{Demographic Ground Truth Data Collection}
Currently in standard clinical practice and research, the most accurate evaluations of cognitive health assessment are one-to-one observation and supervision tasks/questionnaires for monitoring an individual's functional abilities and behavior \cite{resnick15}  In the first stage of this pilot study, we have investigated current literatures and carefully chosen the clinically proven functional and behavioral health assessment survey tools \cite{resnick15}  On the other hand, to cross check with the survey based evaluations, we have also chosen clinically justified observation based behavioral assessment methods  First, following the resident consent, our clinical research evaluator collects demographic and descriptive data (age, gender, race, ethnicity, marital status, education and medical commodities)  She has performed two types of clinical assessments: (1) \emph{Observation based} where the resident's cognition is assessed using the Saint Louis University Mental Status (SLUMS) scale \cite{wai03}  (2) \emph{Survey based} where five widely used and clinically well validated surveys are taken into account: (a) \emph{Yale Physical Activity Survey} \cite{starling99}; (b) \emph{Lawton Instrumental Activities of Daily Living}; (c) \emph{Barthel Index of Activities of Daily Living} \cite{krapp07}; (d) \emph{Geriatric Depression Rating scale} \cite{yesavage82}; and (e) \emph{Zung Self-Rating Anxiety scale} \cite{zung71} 
\subsection{Smart Environment Creation}
For an ideal IoT-based system, instrumenting and deploying it at each participant's natural living environment warrants for assembling a flexible set of hardware and software interfaces to ease the system configuration, setup, and network discovery processes  The sensor system placed in the residences of volunteers needs to meet several specific physiological signals and activity monitoring needs  However, we must confirm that the devices are reliable with potential for re-deployment as well as appear unintimidating to the participants  Inspired by the above requirements, we developed a real testbed IoT system, {\it SenseBox}, by customizing Cloud Engine PogoPlug Mobile base station firmware to integrate with WiFi (connect ambient and object sensors) and Bluetooth (connect wristband) protocol  The smart home components are as follows: (i) PogoPlug base server with a continuous power supply, (ii) 3 binary Passive Infrared sensors in three different rooms (kitchen, livingroom and bedroom) to capture room level occupancy, (iii) 7 binary object sensors attached with closet door, entry door, telephone, broom, laundry basket, trash can and trash box, (iv) three IP cameras in the appropriate positions to collect the ground truth data and (v) an Empatica E4 \cite{empatica} wrist-band (integrated sensors: PPG at 64 Hz, EDA at 4 Hz, Body temperature at 1 Hz and a triaxial ACC at 32 Hz) on the participant's dominating hand 
\section{Activity Recognition}
We aim to detect single wrist-worn ACC sensor based hand gesture and postural activities and insert these into an HDBN graphical model in conjunction with ambient and object sensor values for complex activity recognition  We consider the recognition problem asan activity tupple of $\langle gesture,posture,ambient,object \rangle$  Though, Alam et  al  provides significant performance improvement for single wrist-worn ACC sensor aided 18-hand gesture based postural activity recognition in lab environment \cite{alam17}, it faces some practical challenges in real-time smart environment with older adults due to the diversity of their postures  For example, some older adults use walker, double walking sticks or wheel chair for walking in which cases collecting 18 hand gestures and corresponding postural activities for training requires endless efforts and carefulness  To reduce the complexity of ground truth labeling and later state space explosion for graphical model (HDBN), we propose to use rotational normalization method that can merge some hand-gestures subject to directional differences and forms an 8-hand gesture model  However, our proposed Feature Weight Naive Bayes (FWNB) classifier adds significant improvement on Alam et  al  proposed sparse-deconvolution method as well as recognition in diverse postural environment 
\begin{figure}[!htb]
\begin{center}
   \epsfig{file=hand_gestures pdf,height=0 5in, width=3in}
   \vspace{- 2in}
\caption{8 hand gesture dictionary with direction}
   \label{fig:hand_gestures}
   \vspace{- 2in}
\end{center}
\end{figure}
\subsection{Hand Gesture Recognition}
\label{sec:hand_gesture}
\emph{AutoCogniSys} proposes an 8-gesture dictionary (as shown in Fig  \ref{fig:hand_gestures}) and a Feature Weighted Naive Bayesian (FWNB) framework for building, modeling and recognizing hand gestures  The method comprises of the following steps: (i) \emph{Preprocessing:} wrist-worn ACC sensor provided 3-axis data are passed through 0 4Hz low-pass filter to remove the data drift  (ii) \emph{Rotation normalization:} Normalizing the rotation of hand gestures provides greater accuracy and allows for more realistic, orientation-independent motion  At first, we find the best fit plane of the acceleration vectors thus if the motion lies in a single plane, then the acceleration vectors of a closed shape should on average lie in that main plane  Then, we take all acceleration segments between points of inflection to form one single vector called reference vector that provides us the general direction of user's motion  After that, each vector is normalized relative to the reference vector  This normalization helps remove a lot of hand gestures from prior considered 18 hand gestures resulting a reduced dictionary of 8 gestures  (iii) \emph{Feature Weighted Naive Bayesian model:} Naive Bayes classifier is light-weight and efficient technique for hand gesture recognition  We extract 12 ACC features \cite{alam17} and calculate weight for each feature type based on the similarity of feature measures of the trained gestures (0$<$weight$<$1)  While recognizing gestures, the proximity of each feature measure to the average trained feature measure of each gesture type is calculated by a normal distribution  Then, the proximity value is multiplied by the feature weight that was calculated in the training phase   All of these multiplied values are added together and the system predicts the gesture type with the greatest value as the user gesture  In the learning data points, there should be static postural activities (such as sitting, lying etc ) to avoid unexpected noises over wrist-worn ACC sensors  In the final hand gesture dictionary, we save the reference vector as our signal dictionary 
\subsection{Postural Activity Recognition}
In normal lab environment, wrist-worn ACC sensor signal is a mixture (convolution) of actual hand gesture and postural activity relevant signals \cite{alam17}  \emph{AutoCogniSys} improves the idea by reducing the number of hand gestures and postural activities to 8 (as shown in Fig \ref{fig:hand_gestures}) using rotation normalization and 4 (walking, sitting, standing and lying)  Then, we use sparse-deconvolution method (with 31\% signal reconstruction error) to get Approximately Sparse Factor  The summary of the entire process is stated bellow:

{\it Building Deconvolution Method:} We first consider the wrist-worn ACC sensor signals (3-axis values) as a convolution of hand gesture and postural activity effects and build a deconvolution framework  The deconvolution framework takes a known signal (hand gesture effects) and a equalizer parameter ($\lambda$) as input and provides an Approximately Sparse Factor signal (postural activity effects) as output  For 3-axis ACC signals, we need to learn associated 3 equalizer parameters for each hand gesture  Moreover, each equalizer parameter is involved with 4 postural activities that results a total 96 ($8\times 3\times 4$) equalizer parameters to learn  

{\it Learning Classification Model:} We use the Approximately Sparse Factor signal to extract 12 statistical features and SVM with sequential machine optimization (SMO) \cite{cao06} for postural activity recognition 

{\it Prediction Model:} After recognizing the hand gestures following the method explained in Sec ~\ref{sec:hand_gesture}, we take the corresponding reference vector as known signal and extract the Approximately Sparse Factor signals incorporating corresponding 3 equalizer parameters ($\lambda$) for the sparse-deconvolution method  Then, we apply feature extraction and prior learned SMO based SVM classifier \cite{cao06} to classify final postural activity  Fig ~\ref{fig:deconvolution} illustrates a single axis example of the deconvolution 

\begin{figure}[!htb]
\begin{center}

   \epsfig{file=deconvolution pdf,height=1 6in, width=3in}
   \vspace{- 15in}
\caption{Sample deconvolution example of X-axis  The raw x-axis of accelerometer signal, reference vector of the sample gesture and the extracted corresponding ASF signal of walking }
   \label{fig:deconvolution}
\end{center}
\vspace{- 15in}
\end{figure}

\subsection{Complex Activity Recognition}
We build a HDBN based complex activity recognition framework for single inhabitant scenario smart home environment \cite{alam16b} taking the advantage of detected hand gestural and postural activities along with the ambient and object sensor streams  At first, we obtain instant hand gestural and postural activities from our above proposed models, and additionally motion sensor and object sensor readings from our IoT-system for every time instant generating a 4-hierarchy of HDBN model  Considering the context set $\langle gestural, postural, ambient,object\rangle$ as a hierarchical activity structure (extending two 2-hierarchical HDBN \cite{alam16b}), we build complex activity recognition model for single inhabitant scenario  Finally, we infer the most-likely sequence of complex activities (and their time boundaries), utilizing the well-known Expectation Maximization (EM) algorithm \cite{dempster77} for training and the Viterbi algorithm \cite{forney73} for run-time inference 
\section{Automatic Activity Features Estimation}
The effects of cognitive ability on daily activity performance have been studied before \cite{dawadi14,akl15}  They experimentally and clinically validated that cognitive impairment highly reduces the daily activity performances and this activity performance can be computed as an indicator of cognitive ability status of older adults  The standard activity features refer to completeness of task (TC), sequential task ability (SEQ), interruption avoidance capabilities (INT) etc  In current behavioral science literature, the above activity features carry specific definition based on the sub-tasks involved with a complex activity \cite{dawadi14,akl15}  Completeness of task refers to how many sub-tasks are missed by the participants  Sequential task ability refers to how many sequences of sub-tasks are missed referring the gerontologist defined standard sequences of the sub-task for the particular complex activity  Interruption avoidance capability refers to how many times the participants stop or interleave while doing any sub-task  The final goal of activity features estimation is to provide overall task score  The task score is proportional to the functional ability of participants in performance daily activities  Our behavioral scientist team, comprises with Nursing professor, gerontologist and retirement community caregivers, carefully discus, optimize and choose 87 sub-tasks in total for 13 complex activities 

Each of the sub-task comprises with sequential occurrences of hand gesture and postural activities  However, no researchers ever considered hand gesture for activity features estimation due to complexity of multi-modal wearable and ambient sensors synchronization and multi-label activity classification \cite{dawadi14,akl15}  \emph{AutoCogniSys} exploited single wrist-worn sensor based hand gesture and postural activity recognition, and proposed an activity features (TC, SEQ and INT) estimation method including these two parameters in conjunction with object and ambient sensor features that provide significant improvement of cognitive health assessment of older adults 
\subsection{Machine Learning Based Complex Activity Features Estimation}
In current cognitive health assessment literature, complex activity features can be defined as $\langle TC,SEQ,INT,TS\rangle$  We used supervised method to estimate TC, SEQ and INT, and unsupervised method to estimate TS  We first, formulate the automated scoring as a supervised  machine learning problem in which machine learning algorithms learn a function that maps $\langle${\it hand gesture, posture, object, ambient sensor}$\rangle$ feature set to the direct observation scores  We use bagging ensemble method to learn the mapping function and SMO based SVM \cite{cao06} as base classifier  The learner averages by boostrapping individual numeric predictions to combine the base classifier predictions and generates an output for each data point that corresponds to the highest-probability label  We train three classifiers considering observation as ground truth for TC, SEQ and INT scores and test on the testing dataset  We derive unsupervised scores using dimensionality reduction technique for each feature set  First, we take all features of each activity, apply optimal discriminant analysis technique as a dimensionality reduction process \cite{zhang09} and reduce the feature sets into single dimensional value which represents the automated task completeness scores of the particular user activity  A min-max normalization is applied that provides us a uniform range of the variables using $
z_i=\frac{x_i-min(x)}{max(x)-min(x)}$ equation where $x=\{x1,\ldots,x_n\}$ and $z_i$ is $i^{th}$ normalized data  The final single dimensional score represents machine learning based TS score 
\section{Physiological Sensor Signals Processing}
The autonomic nervous system (ANS) restrains the body's physiological activities including the heart rate, skin gland secretion, blood pressure, and respiration  The ANS is divided into sympathetic (SNS) and parasympathetic (PNS) branches  While SNS actuates the body's resources for action under arousal conditions, PNS attenuates the body to help regain the steady state  Mental arousal (say stress, anxiety etc ) activates the sweat gland causing the increment and reduction of  Skin Conductance on SNS and PNS physiological conditions respectively  However, Instant Heart Rate also has similar effect on SNS and PNS physiological condtions i e , a higher value of heart rate is the effect of SNS and lower value is the outcome of PNS  EDA and PPG sensors are widely used to estimate the instant value of skin conductance and heart rate respectively \cite{alam16} 
\subsection{EDA Sensor Signal Processing}
EDA is the property of the human body that causes continuous variation in the electrical characteristics of the skin which varies with the state of sweat glands in the skin  There are three types of arousal: \emph{cognitive, affective and physical}  \emph{Cognitive} arousal occurs when a person tries to solve any problem using her cognitive ability  \emph{Affective} arousal occurs when a person is worried, frightened or angry either doing daily activities or in resting position  On the other hand, \emph{physical} arousal is related to the brain command to move bodily parts which is imposed on the total arousal as an artifact, called \emph{motion artifact}  However, there are always some noises due to the weather conditions (temperature, humidity etc ) and device motion  This \emph{motion artifact} can be the prime cause of signal contamination of physiological outcomes while performing daily activities which must be removed  \emph{AutoCogniSys} proposes an EDA sensor signal processing method consists of three steps: (i) noise and motion artifacts removal, (ii) separation of tonic component and phasic component (explained later) from contamination free EDA signal and (iii) feature extraction on the response window 
\subsubsection{Motion Artifacts Removal}
There are many types of motion artifacts but the unsual steep rise is the mostly occured ones associated with EDA signal while performing daily activities \cite{edel67}  We use well-known steep rising noises reduction technique, SWT \cite{chen15}  We first consider EDA signal as a mixture of a slow variant tonic and fast variant phasic component, i e , SWT coefficient is modeled as a mixture of two Gaussian components, phasic (close to zero valued signal) and tonic (high rising signal)  After expanding EDA signal into multiple levels of scaling and wavelet coefficients, we choose adaptively a threshold limit at each level based on the statistical estimation of the wavelet coefficients' distribution, and employ that on the wavelet coefficients of all levels  Finally, an inverse wavelet transform technique is applied to the thresholded wavelet coefficients to obtain the artifacts free EDA signal  Fig~ \ref{fig:eda_artifact_removal} shows a sample of raw and motion artifacts free EDA signal 

\begin{figure}[!htb]
\begin{center}
\vspace{- 1in}
   \epsfig{file=eda_signal_artifact pdf,height=1 6in, width=3 5in}
\caption{Dashed line represents noisy EDA signal and solid red line represents \emph{AutoCogniSys} proposed motion artifact free EDA signal}
   \label{fig:eda_artifact_removal}
\end{center}
\end{figure}
\subsubsection{Convex Optimization Technique to EDA Deconvolution}
After the motion artifact removal, we consider EDA as the sum of three components for $N$ sample: a slow tonic driver ($t$), fast (compact, bursty) non-negative sparse phasic driver ($r$) and a reminder error term ($\epsilon_r$) 
\begin{equation}
\label{eq:eda_signal}
y = t + r + \epsilon_r
\end{equation}
This additive error $\epsilon_r$ is a White Gaussian Noise  The central problem associated with the deconvolution method is to get tonic $t$ component from the above equation  \cite{greco16} showed that EDA signal deconvolution (separation of tonic, phasic and noise terms from EDA signal) is a quadratic optimization problem and defined tonic component as follows:
\begin{equation}
\label{eq:tonic}
t = Bl + Cd,
\end{equation}
where $B$ is a tall matrix whose columns are cubic $B$-spline basis functions, $l$ is the vector of spline coefficients, $C$ is a $N\times 2$ matrix, $d$ is a $2\times 1$ vector with the offset and slope coefficients for the linear trend  The above equation is subject to the following optimization problem,
\begin{eqnarray}
minimize \frac{1}{2} {||Mq + Bl + Cd- y||}^2_2 +\alpha {||Aq||}_1 + \frac{\lambda}{2} {||l||}^2_2\\
subject\;to\; Aq \geq 0\nonumber
\end{eqnarray}
where $M$ and $A$ are tridiagonal matrices and $q$ is an auxiliary variable  After solving the above equation, we can get the optimal values for $\{q,l,d\}$ that can be used to obtain tonic component from the equation~\ref{eq:tonic}  The reminder of the equation~\ref{eq:eda_signal} ($r+\epsilon_r$) is considered as a mixture of White Gaussian Noise ($\epsilon_r$) and a fast variant phasic component ($r$)  We employ butterworth low-pass filter (5Hz) and hanning smoothing with window size 4 (optimal) to remove $\epsilon_r$ from phasic component ($r$) 
\subsection{PPG Signal Processing}
PPG is used mainly for measuring the oxygen saturation in the blood and blood volume changes in skin  An ideal PPG signal processing must contain the following steps: noise and motion artifacts removal, heart rate detection, heart rate variability estimation and feature extraction 
\subsubsection{PPG Signal Noise and Motion Artifacts Removal}
Similar to EDA signal, PPG signal is also contaminated with motion artifacts and noises  However, unlike EDA signal, PPG produce quasiperiodicity in a time series spectrum \cite{mete30}  We use Periodic Moving Average Filter (PMAF) to remove motion artifacts and noises \cite{lee07}  We first segment the PPG signal on periodic boundaries and then average the $m^{th}$ samples of each period  After filtering the input PPG signal with a 5-Hz $8^{th}$-order Butterworth low-pass filter, we estimate the maximum and minimum value of each period  The mean of each period are obtained from the maximum and minimum values applying the zero crossing method  These points of the means help determine the boundaries of each period  Then, interpolation or decimation is performed to ensure that each period had the same number of samples \cite{lee07}  
\subsubsection{Heart Rate and Heart Rate Variability Estimation}
We first apply PMAF on PPG signal to remove noises and motion artifacts, refine PPG by smoothing the signal using 1-dimensional Gaussian Filter and Convolution, calculate first derivative of the convoluted signal and finally find the differences between two consecutive peak values which is called HRV \cite{sel08}  The occurrences of total peak values (R-peak or beat) in each minute is called Heart Rate (HR) with an unit of Beat Per Minute  The signal value property of HRV and HR are inversely proportional which means the mental arousal that increases HR should decrease HRV in the time segment window  Fig~\ref{fig:ppg_artifact_removal} shows a sample of the noisy and filtered PPG signal and their corresponding Instant Heart Rate 
\begin{figure}[!htb]
\vspace{- 1in}
\begin{center}
   \epsfig{file=ppg_artifact_removal pdf,height=1 4in, width=3 5in}
   \vspace{- 15in}
\caption{Top figure illustrates the noisy signal (dotted line) and filtered signal from PPG sensor based on our filtering method  Bottom figure illustrates instant heart rate calculated from noisy signal (dotted line) and filtered signal}
   \label{fig:ppg_artifact_removal}
\end{center}
\vspace{- 15in}
\end{figure}
\subsection{Physiological Sensor Signal Feature Extraction}
Using the above mentioned methods, we removed the noises and motion artifacts from EDA and PPG signals and generated two time series signal from EDA (tonic and phasic components) and one time series signal from PPG (HRV)  Then, we segment each of the time series signal based on our prior detected complex activities such that each response window starts and ends with the starting and ending points of each complex activity  We extract 7 statistical time-series features for EDA (as shown in Table~\ref{tab:eda_features}) and 8 features for HRV (Table~\ref{tab:hrv_features}) within the response window) 

\begin{table}[!t]
\begin{center}

\renewcommand{\arraystretch}{1}
\caption{EDA Features Within The Response Window}
\begin{scriptsize}


\label{tab:eda_features}
\begin{tabular}{|c|l|}
\hline
\bfseries Features& \bfseries Description\\
\hline
nSCR & Number of SCRs within response window (wrw)\\
\hline
Latency & Response latency of first significant SCR wrw\\
\hline
AmpSum & Sum of SCR-amplitudes of significant SCRs wrw\\
\hline
SCR & Average phasic driver wrw\\
\hline
ISCR & Area (i e  time integral) of phasic driver wrw\\
\hline
PhasicMax & Maximum value of phasic activity wrw\\
\hline
Tonic & Mean tonic activity wrw\\
\hline
\end{tabular}
\end{scriptsize}
\end{center}
\end{table}



\begin{table}[!t]
  \begin{center}
\renewcommand{\arraystretch}{1}
\vspace{- 3in}
\caption{Heart Rate Variability features}
\label{tab:hrv_features}
\begin{scriptsize}
\begin{tabular}{|c|l|}

\hline
\bfseries Feature& \bfseries Description\\
\hline
$\overline{RR}$&Mean RR intervals\\
\hline
SDNN&Standard deviation of RR intervals\\
\hline
SDSD&Std of successive RR interval differences\\
\hline
RMSSD&Root mean square of successive differences\\
\hline
NN50&\#successive intervals differing more than 50 ms\\
\hline
pNN50&relative amount of NN50\\
\hline
HRVTI&Total number of RR intervals/height of the histogram\\
\hline
TINN&Width of RR histogram through triangular interpolation\\
\hline
\end{tabular}
\end{scriptsize}
  \end{center}
\end{table}
\section{Experimental Evaluation}
In this section, we explain our data collection, available benchmark dataset, baseline methods and evaluation 
\subsection{Datasets and Baseline Methods}
We validate and compare \emph{AutoCogniSys} with baseline methods on both publicly available and our collected datasets 
\subsubsection{RCC Dataset: Collection and Ground Truth Annotation}
For collecting Retirement Community Center Dataset (RCC Dataset), we recruited 22 participants (19 females and 3 males) with age range from 77-93 (mean 85 5, std 3 92) in a continuing care retirement community with the appropriate institutional IRB approval and signed consent  The gender diversity in the recruited participants reflects the gender distribution (85\% female and 15\% male) in the retirement community facility  A trained gerontology graduate student evaluator completes surveys with participants to fill out the surveys  Participants are given a wrist band to wear on their dominant hand, and concurrently another trained IT graduate student have the IoT system setup in participants' own living environment (setup time 15-30 minutes)  The participants are instructed to perform 13 \emph{complex ADLs}  Another project member remotely monitors the sensor readings, videos and system failure status  The entire session lasts from 2-4 hours of time depending on participants' physical and cognitive ability 

We follow the standard protocol to annotate demographics and activities mentioned in the IRB  Two graduate students are engaged to annotate activities (postural, gestural and complex activity) whereas the observed activity performances are computed by the evaluator  Two more graduate students are engaged to validate the annotations on the videos  In overall, we are able to annotate 13 complex activities (total 291 samples) labeling for each participant; 8 hand gestures (total 43561 samples) and 4 postural activities (total 43561 samples) labeling  Annotation of postural and complex activities outcomes no difficulties from recorded videos  However, annotation of hand-gestures is extremely difficult in our scenario  We used video based hand tracker that can track and sketch wrist movements from a video episode \cite{hugo14}  This sketching can help us significantly to identify which particular hand gesture is being performed in the time segment 
\subsubsection{EES Datasets: EDA and PPG Sensor Datasets}
We used Eight-Emotion Sentics (EES) dataset to validate \emph{AutoCogniSys} proposed physiological signal processing approaches \cite{picard01}  The dataset consists of measurements of four physiological signals (PPG/Blood Volume Pulse, electromyogram, respiration and Skin Conductance/EDA) and eight affective states (neutral, anger, hate, grief, love, romantic love, joy, and reverence)  The study was taken once a day in a session lasting around 25 minutes for 20 days of recordings from an individual participant  We consider only PPG and EDA for all of the affective states in our study 
\subsubsection{Baseline Methods}
Though no frameworks ever combined all modalities together into real-time automated cognitive health assessment, we evaluate \emph{AutoCogniSys} performance by comparing the performances of its components individually with upto date relevant works  For hand gesture and postural activity recognition, we consider \cite{alam17} proposed method as baseline  For complex activity recognition, we compare our hand gesture and postural activity classifiers aided HDBN model with three-level Dynamic Bayesian Network \cite{zhu12} framework  For activity performance estimation, activity performance based cognitive health assessment; and EDA and PPG based cognitive health assessment, we have considered \cite{alam16} proposed method as baseline 
\subsection{Activity Recognition Evaluation}
The standard definition for \emph{accuracy} in any classification problem is $\frac{TP+TN}{TP+TN+FP+FN}$ where $TP,TN,FP$ and $FN$ are defined as true positive, true negative, false positive and false negative  For complex activity recognition evaluation, we additionally consider \emph{start/end duration error} as performance metric that can be explained as follows: consider that the true duration of ``cooking'' is 30 minutes (10:05 AM - 10:35 AM) and our algorithm predicts 29 minutes (10 10 - to 10 39 AM)  Then, the start/end duration error is 9 minutes ($|$5 minutes delayed start$|$ + $|$4 minutes hastened end$|$), in an overall error of e g , 30\% (9/30=0 3)  We measure cross-participant accuracy using leave-two-participants-out method for performance metrics, i e , we take out two of the participants' data points from the entire dataset, train our proposed classification models, test the model accuracy on the two left-out participants relevant data points, and continue the process for entire dataset 

\begin{figure*}[!htb]
\begin{minipage}{0 45\textwidth}
\begin{center}
   \epsfig{file=hand_gesture_accuracy pdf,height=1 6in, width=3in}
\caption{Feature Weighted Naive Bayes (FWNB) classification accuracy comparisons with baseline approaches (graphical signatures of all hand gestures are shown) }
   \label{fig:hand_gesture_accuracy}
\end{center}
\end{minipage}
\begin{minipage}{0 29\textwidth}
\begin{center}
\vspace{- 12in}
   \epsfig{file=posture_accuracy_normal pdf,height=1 6in, width=2 1in}
\caption{4-class postural level activity recognition performance and comparisons with baseline method}
   \label{fig:posture_accuracy_normal}
\end{center}
\end{minipage}
\begin{minipage}{0 25\textwidth}
 \begin{center}
 \vspace{- 12in}
   \epsfig{file=posture_accuracy_extended pdf,height=1 6in, width=2 1in}
\caption{6-class diverse postural activity recognition framework accuracy comparisons with the baseline approach }
   \label{fig:posture_accuracy_extended}
\end{center}
 \end{minipage}
\end{figure*}

Fig~\ref{fig:hand_gesture_accuracy} displays Feature Weighted Naive Bayes (FWNB) based the 8-hand gestural activity recognition accuracies comparisons with the baseline methods which clearly depicts the outperformance of our method (5\% improvement) with an overall accuracy of 92\% (FP rate 6 7\%) in RCC dataset  For postural activity recognition, dataset achieving 91\% postural activity recognition accuracy (FP rate 9 5\%) which outperforms the baseline approach significantly (8\% improvement)  Now, we expand the postural activities for RCC datasets into 3 diverse `walking' postures: `normal walking', `walking with walker', `walking with single stick' and the accuracy goes down to 88\% (FP 7 9\%)  Fig ~\ref{fig:posture_accuracy_normal} and Fig ~\ref{fig:posture_accuracy_extended} illustrate 4-class postural and extended 6-class postural classifier accuracies respectively which clearly posit that \emph{AutoCogniSys} outperforms in each case of postural activities as well as overall performances (8\% and 7\% improvement respectively) 

For complex activity classification, we choose RCC dataset to train our HDBN model  Our leave-two-participants out method results an accuracy of 85\% (FP Rate 3 6\%, precision 84 2\%, recall 84 5\%, ROC Area 98 2\%) with a start/end duration error of 9 7\%  We run the entire evaluation for baseline complex activity recognition algorithm too achieving an overall accuracy of 78\% (FP Rate 5 2\%, precision 79 6\%, recall 78 5\%, ROC Area 82 7\%) which is clearly lower performed method than our approach  Fig  \ref{fig:complex_activity_roc} and Fig~\ref{fig:complex_activity_accuracy} illustrate the ROC curve and each complex activity recognition accuracy comparisons with baseline method which depict the outperformance of our framework over baseline methods (7\% improvement)  Fig~\ref{fig:complex_activity_accuracy} also shows that inclusion of postural activity improves the final complex activity recognition (4\% improvement) 
 \begin{figure}  [!htb]
  \begin{minipage}{0 15\textwidth}
 \begin{center}
   \epsfig{file=complex_activity_roc pdf,height=1 4in, width=1 1in}
\caption{ROC curve for complex activity recognition}
   \label{fig:complex_activity_roc}
\end{center}
\end{minipage}
\begin{minipage}{0 33\textwidth}
\begin{center}

   \epsfig{file=complex_activity_accuracy pdf,height=1 4in, width=2 3in}
\caption{Complex ADLs recognition accuracy improvement and comparison with baseline \cite{zhu12} and HMM based method}
   \label{fig:complex_activity_accuracy}
\end{center}

\end{minipage}
\end{figure}

\subsection{Quantification of Performance Score}
To characterize both the qualitative and quantitative health assessment performance scores, we start with four different feature groups ranging from both functional and physiological health measures: (i) observation based activity features, (ii) automatic activity performance features, (iii) EDA features and (iv) PPG features 

In \emph{observation based activity features}, we design a complex activity set comprised of multiple subtasks which are involved with task {\it interruption, completion and sequencing}  Participants are instructed to perform the complex activities while the trained evaluator observed the aforementioned functional activity performance measures  Each incorrect attempt of performance measure will be assigned one point thus higher score reflects lower performance of functional activities \cite{dawadi14}  We first detect hand gesture and postural activities  Then, we feed the low-level activity contexts (gestural and postural) combined with ambient contexts (object and ambient motion sensor readings) into HDBN for single inhabitant model \cite{alam16b} to recognize complex activities  The complex activity recognition framework provides both activity labels and activity window (start-end points)  Then, we extract features of object sensor, ambient sensor, gestural activity and postural activity events for each activity window  The features are number of occurrences, mean number of occurrences, consecutive 1, 2, 3, $\ldots$ 20 occurrences, top 10, 20, 30, $\ldots$, 90 percentile etc (29 features in total)  In \emph{physiological features} we first detect 13 complex activities using HDBN algorithm which provides activity labels and activity window (start-end points), apply noise reduction, motion artifacts removal, extract 7 EDA features and 8 HRV features for each activity and take the mean of them over time (minutes) to get 15 (7+8) complex activity physiological features set for each participant  In summary, we extract 3 observation based activity features, 29 automatic activity performance features, 7 EDA features and 8 HRV features 
\subsection{Physiological Signal Processing Performance Evaluation}
Standard evaluation technique should use both experimental and publicly available datasets to confirm the outperformance of the novel approaches  We first evaluate our physiological signal processing techniques using a publicly available dataset (EES Dataset \cite{picard01}) to detect 8 human emotions  Then, in next section, we evaluate our methods in assessing cognitive health status of older adults using RCC dataset 

For EDA, we first apply SWT method to remove motion artifacts and noises  Then, we use cvxEDA method to separate tonic and phasic components of EDA  Then, we extract 7 EDA features on a sliding window of 4 seconds  Finally, we feed the 7 EDA features into a SMO based SVM algorithm \cite{cao06}  We use 10-fold cross validation to classify eight emotions achieving 87\% of overall accuracy (FP rate 6\%)  For PPG, we first apply our proposed PMAF based noises and motion artifacts removal technique  Then, we calculate HRV and perform time-domain feature extraction to extract 8 HRV features on a sliding window of 4 seconds  We feed these features into a SMO based SVM algorithm \cite{cao06}  Our 10-fold cross validation shows accuracy of 79\% (FP rate 11 5\%) of detecting 8 emotions on EES Dataset  Fig  \ref{fig:ees_eda} and Fig  \ref{fig:ees_ppg} clearly depict that \emph{AutoCogniSys} proposed EDA and PPG signal processing techniques significantly improve the accuracy over the baseline \cite{alam16} method (10\% and 12\% improvement) 

\begin{figure}[!htb]
\begin{minipage}{0 24\textwidth}
\begin{center}
   \epsfig{file=ees_eda pdf,height=1 2in, width=1 8in}
\caption{(EES Databaset) EDA features based Eight Emotion classification accuracy comparisons with baseline method}
   \label{fig:ees_eda}
\end{center}
\end{minipage}
\begin{minipage}{0 23\textwidth}
\begin{center}
   \epsfig{file=ees_ppg pdf,height=1 2in, width=1 7in}
\caption{(EES Dataset) PPG features based 8-Emotion classification accuracy comparisons with baseline method}
   \label{fig:ees_ppg}
\end{center}
\end{minipage}

\end{figure}
\subsection{Evaluation of Performance Scores}
The feature subsets used in the experimentation for observation and survey based clinical assessments and technology guided physiological and activity initiated health assessments are depicted in Table~\ref{tab:feature_subset}  From our 6 demographics surveys, we find significant distributions in terms of cognition only for SLUMS Score (S-Score)  Based on that, we divide our participants pool into three groups: \emph{Not Cognitively Impaired (NCI), Mild Cognitively Impaired (MCI) and Cognitively Impaired (CI)} where the number of participants are $5$, $7$ and $10$ respectively 
\begin{table}[!t]
\begin{scriptsize}


{\centering 
\renewcommand{\arraystretch}{ 6}
\caption{Feature Subsets}
\label{tab:feature_subset}
\begin{tabular}{|l|L{5 5cm}|}
\hline
\bfseries Feature& \bfseries Description\\
\hline
Observation & Task Completeness (TC), Sequencing (SEQ), Interruptions (INT)\\
\hline
Survey & SLUMS Score (S-Score), ZUNG Score (Z-Score), IADL Score (I-Score), Yale Score (YPAS), Barthel Score (B-Score), GDS Score (G-Score)\\
\hline
EDA  and HRV & 7 and 8 Features\\
\hline
Activity Performance& Supervised (TC, SEQ, INT), Unsupervised\\
\hline
Arousal& EDA and HRV features of each complex activity window\\
\hline

\end{tabular}
}
\end{scriptsize}
\end{table}


\begin{figure}[!htb]
\begin{center}
   \epsfig{file=group_correlation pdf,height=1in, width=3 3in}
\caption{\emph{AutoCogniSys} Proposed Method Based Group Correlation analysis ( $r-value$) NCI, MCI and CI represent not cognitive, mild cognitive and cognitively impaired group of population  TC, INT, SEQ, EDA and HRV represent task completeness, interruption scores, sequencing scores, electrodermal activity features and heart rate variability features}
   \label{fig:group_correlation}
\end{center}
\vspace{- 2in}
\end{figure}
\begin{figure}[!htb]
\begin{center}
   \epsfig{file=group_correlation_baseline pdf,height=1in, width=3 3in}
\caption{Baseline \cite{alam16} method based Group Correlation analysis ( $r-value$)}
   \label{fig:group_correlation_baseline}
   \vspace{- 25in}
\end{center}
\end{figure}

\subsection{Statistical Correlation Analysis of Cognitive Health}
We used Pearson correlation coefficients with significance on $p<0 05$* for individual feature and partial correlation coefficients with significance on $p<0 005$** for group of features correlation analysis  Fig  \ref{fig:group_correlation} and Fig  \ref{fig:group_correlation_baseline} show the group correlation analysis results based on \emph{AutoCogniSys} proposed framework and baseline \cite{alam16} framework respectively  It can be clearly depicted that our proposed framework improves the correlation with the ground truths 
\subsection{Machine Learning Classification of Cognitive Health}
We evaluate using machine learning classifiers to predict cognitive status of older adults using both individual modalities and combined features  We use leave-two-participants out method to train and test classification accuracy 

We first choose the individual activity features (machine learning method based interruption scores, sequencing scores, unsupervised scores) and their combined features to train and test cognitive impairment status classification for SMO based SVM algorithm \cite{cao06}  The classification accuracies are 72\%, 69\%, 76\% and 83\% respectively  Then we consider 7 EDA-activity features and 8 HRV-activity features individually in training and testing phase of SMO based SVM algorithm \cite{cao06} resulting 85\% and 80\% accuracy respectively 

\begin{figure}[!htb]
\begin{minipage}{0 24\textwidth}
\begin{center}
   \epsfig{file=combined_classification pdf,height=1 2in, width=1 7in}
   \vspace{- 15in}
\caption{Individual and combined classification accuracies comparison with baseline method for cognitive impairment status detection}
   \label{fig:combined_classification}
\end{center}
\end{minipage}
\begin{minipage}{0 23\textwidth}
\begin{center}
   \epsfig{file=each_activity_cognitive_assessment pdf,height=1 2in, width=1 7in}

\caption{Machine learning based cognitive health assessment accuracy for each complex activity in terms of activity, EDA and HRV features }
   \label{fig:each_activity_cognitive_assessment}
\end{center}
\end{minipage}
\end{figure}

For combined classifier, we first applied sequential forward feature selection to find the best combinations of 1- 3 features for cognitive impairment classification group MCI, NCI and CI in terms of combined activity features (29 features), EDA-activity features (7 features) and HRV-activity features (8) features  Our final combined classifier (SMO based SVM algorithm \cite{cao06}) provides an accuracy of {\bf 93\%} in detecting the cognitive impairment status of older adults  Fig  \ref{fig:combined_classification} shows our proposed individual and combined methods outperform the baseline \cite{alam16} significantly (13\% improvement)  Fig  \ref{fig:each_activity_cognitive_assessment} shows the cognitive impairment status prediction accuracy for each modality (activity feature, EDA and HRV) per individual complex activity 
\subsection{Discussion}
If we exclude the postural activities from automated activity performance scoring, we find reduced statistical correlation with original task completeness performance for \{NCI, MCI, CI\} participant group (INT 0 53*, SEQ 0 21' and unsupervised 0 49')  However, if we skip our proposed motion artifact removal stage, we find reduced statistical correlation with \{NCI, MCI\} and \{MCI, CI\} groups of participants (EDA and HRV correlations respectively \{0 51*, -0 51*\} and \{-0 53*,0 46\})  To test our proposed motion artifacts removal impact on EDA signals more rigorously, we choose 5 random participants, engage one expert motion artifact annotator to annotate motion artifacts segment on each participant's first 30 minutes of complex dataset using recorded video and apply both baseline and our methods to detect motion artifact segments  While baseline method achieves 75 5\% (FP rate 20 3\%) accuracy in detecting motion artifact segments, \emph{AutoCogniSys} outperforms achieving 89 9\% (FP rate 8 9\%) accuracy  In terms of experience, we have seen 100\% acceptance of wearing wrist-band,  71\% of acceptance for signing consent on using cameras and 0\% failure rate of collecting continuous data 
\section{Conclusion}
We propose, \emph{AutoCogniSys}, an IoT inspired design approach combining wearable and ambient sensors embedded smart home design, extensive signal processing, machine learning algorithms and statistical analytics to automate cognitive health assessment in terms of complex activity performances and physiological responses of daily events  Additionally, our postural activity detection approach in diverse population cum improved activity performance measurement and fundamental physiological sensor artifacts removal from physiological sensors help facilitate the automated cross-sectional cognitive health assessment of the older adults  Our efficient evaluation on each modality (physical, physiological and ambient) and each activity mode proves that any of the mode (say single activity and single sensor) also can provide significant improved cognitive health assessment measure 







ground_truth: SNNs have the potential to better model and explain the functional hierarchy and mechanisms of the visual system.

prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: What types of sensors are now capable of estimating physical activity levels and physiological outcomes of older adults?

Read the following text and answer briefly.

Context 3: A Brief History of Benjamin Franklin's Residences on Craven Street, London: 1757 - 1775 - Journal of the American Revolution
Benjamin Franklin House, 36 Craven St, London  (Photo by Elliott Brown | Wikimedia Commons)
If one looked into Benjamin Franklin’s time on Craven Street, they might initially believe he lived at 36 Craven Street the entirety of his two stays in London based on the plethora of articles on the internet that say so  If they dug a little deeper they might read that he lived at No  27 Craven Street, previously numbered 7, but now numbered 36; or that he lived exclusively at No  7 Craven Street; or that he lived in multiple residences on Craven Street; or that he moved out of No  36 to another house on Craven Street and then moved back into No  36 the last year of his residence  What is one to believe with all of the conflicting accounts? What does the historical record have to say about Franklin’s time on Craven Street?
Figure 1  Spur Alley 1685  “A map of the parish of St Martins in the Fields, taken from ye last survey, with additions (1685)”  (© The British Library Board, Shelfmark: Maps Crace Port  13 2, Item number: 2)
Before Craven Street existed there was Spur Alley, a narrow passageway sandwiched between the Hungerford Market to the north (now Charing Cross Station) and Scotland Yard and the Northumberland House and Garden to the south  It was flanked on both ends by major thoroughfares, the Strand on the west, connecting Westminster to London by road, and the River Thames on the east, not only connecting the two cities to each other and to Southwark on the south side of the Thames, but connecting the entire metropolis to the rest of the world  Being located in the City of Westminster, Spur Alley had escaped the devastation of the Great Fire of London in 1666 leaving its wooden structures, built in the early part of seventeenth century, intact, but also in dire need of restoration or demolition  “The ratebooks show that during the last thirty years or so of their existence the houses in Spur Alley were in a very bad condition  Few of them were rated at more than a few shillings and many of them were unoccupied ”[1] The landowner, William, 5th Baron Craven, desiring to increase the profitability of his assets, tore down the derelict structures on Spur Alley around 1730 and leased the newly established lots to builders  By 1735, twenty brick houses in the Georgian style had been built on the west side and sixteen on the east side of the way now called Craven Street [2]
Figure 2  Craven Street 1746  (John Rocque London, Westminster and Southwark, First Edition 1746, Motco Enterprises Limited, motco com)
Letters to Franklin during his residence with Mrs  Margaret Stevenson, his landlady on Craven Street, were addressed rather vaguely; “Craven Street/Strand”, “Mrs  Stevensons in Craven Street”, or “Benjamin Franklin Esqr ” are but a few examples  Letters from Franklin referenced “London,” or sometimes “Cravenstreet,” but never included a number  Despite the absence of numbered addresses in Franklin’s correspondence, there was a sense of one’s place in the neighborhood based on entries in the Westminster Rate Books (tax assessments)  The Rate Books did not list house numbers during Franklin’s time there, but they did list the residents of Craven Street in a particular order that became the default numbering system for the street  Number one was associated with the first resident listed under “Craven Street” in the Rate Books and was the northernmost house on the west side of the street  The numbers increased counter-clockwise down the west side and up the east side in accordance with the list of residents  In 1748, the first year of Margaret Stevenson’s (Stevens in the Rate Books for that year) residence on Craven Street, she is listed as the twenty-seventh resident, the second house north of Court Street (later Craven Court, now Craven Passage) on the east side of the street [3]
In 1766, Parliament passed the London Paving and Lighting Act (6 Geo  3 c  26), “An act for the better paving, cleansing, and enlightening, the city of London, and the liberties thereof; and for preventing obstructions and annoyances within the same; and for other purposes therein mentioned ”[4] One of the other purposes therein mentioned was the numbering of houses  With an aim to bring order to the chaotic numbering systems or lack thereof on London streets the Act provided that “… the said commissioners … may also cause every house, shop, or warehouse, in each of the said streets, lanes, squares, yards, courts, alleys, passages, and places, to be marked or numbered, in such manner as they shall judge most proper for distinguishing the same ”[5] This was quite an undertaking that took years to accomplish  It was a decade later before numbered addresses on Craven Street in the City of Westminster appeared in The London Directory (1776)  The London Directory and its competitors were published primarily by booksellers or printers to supplement their income and were highly profitable  To say they were competitive is an understatement  “Some of the most hotly disputed struggles over copyright in the century concerned guidebooks  Many were optimistically emblazoned with a royal license and a notice that the work had been entered at Stationers’ Hall  Various struggles between rival guides intensified as the potential for profits became clear ”[6] The London Directory boldly proclaimed to contain “An ALPHABETICAL LIST OF THE NAMES and PLACES of ABODE of the MERCHANTS and PRINCIPAL TRADERS of the Cities of LONDON and WESTMINSTER, the Borough of SOUTHWARK, and their Environs, with the Number affixed to each House ”[7] Kent’s Directory made a similar proclamation: “An Alphabetical LIST OF THE Names and Places of Abode OF THE DIRECTORS of COMPANIES, Persons in Public Business, MERCHANTS, and other eminent TRADERS in the Cities of London and Westminster, and Borough of Southwark WITH THE NUMBERS as they are affixed to their Houses agreeable to the late Acts of Parliament ”[8] Mrs  Stevenson wasn’t included in the directories because she didn’t meet the criteria of being a merchant or trader, not because she was a woman  Although it is rare to see women listed in the directories, some examples do exist [9] If Mrs  Stevenson had appeared in the directories in 1776 it would not have been on Craven Street as she had moved to Northumberland Court, a stone’s throw away, the previous year [10] A comparison of Craven Street residents whose names and addresses do appear in the directories with the same residents as they appear in the Westminster Rate Books determines if the numbering systems were congruent  For the most part they were  For example, Joseph Bond at No  30, William Rowles at No  31, Samuel Sneyd at No  32, and Jonathan Michie at No  35 in The London Directory coincide with their places of residence in the Westminster Rate Books; however, errors did occur  The 1776 edition of The London Directory lists Brown & Whiteford, wine merchants, at No  9 Craven Street while the Westminster Rate Books list them as the twenty-ninth residents  Obviously, it makes no sense to have Brown & Whiteford at No  9 in The London Directory and their next-door neighbor, Joseph Bond, at No  30  The same error appears in Baldwin’s The New Complete Guide for 1783  The New Complete Guide may have “borrowed” the error from The London Directory  It was not uncommon for the owner of one directory to copy entries from another to save both time and money  Beginning in 1778 and contrary to The London Directory, Kent’s Directory faithfully followed the numbering system of the Westminster Rate Books in all of its editions and listed Brown & Whiteford at No  29 as did Bailey’s Northern Directory in 1781  Perhaps realizing their error, The London Directory changed their listing of Brown & Whiteford from No  9 to No  29 in their 1783 edition and maintained that listing thereafter 
Sometime prior to 1792, the embankment on the Thames at the south end of Craven Street had been sufficiently extended allowing for the construction of ten new houses below the original houses: “ … four houses, Nos  21–24, were built on the west side, and six houses, Nos  25–30, on the east side of the way ”[11] In a note in the same report, the new numbering system is explained  “The houses in the street, which had previously been numbered consecutively down the west side and up the east side, were then renumbered on the same system to include the additional houses ”[12] Because the new houses (21-24) on the west side were built below the existing houses (1-20), houses 1-20 retained their original numbering 
Figure 4  Craven Street 1799  (Richard Horwood’s Map of London, Westminster and the Borough of Southwark 1799, Motco Enterprises Limited, motco com)
One would think that the numbers of the sixteen original houses on the east side, Nos  21 – 36, would simply increase by ten with the addition of the ten new houses, but such was not the case; they increased by nine  How could that be? The only possible explanation is that No  21 of the original houses was demolished to make way for the construction of the northernmost of the six new houses on the east side (No  30)  Evidence of No  21’s demolition appears in the lease granted to Charles Owen by William, 7th Baron Craven, in 1792, which describes No  22 as: “All that messuage in Craven Street late in the occupation of Francis Deschamps undertaker … being the Southernmost house in the Old Buildings on the East Side of the said Street numbered with the No  22 ”[13] The lease describes No  22 as being the southernmost house in the old buildings on the east side of Craven Street  Clearly the house previously at No  21 did not exist when the lease granted to Charles Owen was written in 1792 as it used to be the southernmost house  It is also worth noting that in 1790, The London Directory listed Jacob Life at No  21 (original numbering)  In 1791-2, it listed him at No  6  With No  21 vacated, it would allow for its demolition and the construction of the tenth new house  By utilizing lot No  21 for the new construction, only nine additional lots were needed to build the ten houses, hence, Margaret Stevenson’s former residence at 27 became 36 (27 + 9) in the renumbering and not 37 
For nearly a century and a half after Franklin departed London for America in March of 1775 the scales were tipped heavily in favor of his residence having been No  7 Craven Street  As early as 1807 in London; Being An Accurate History And Description Of The British Metropolis And Its Neighborhood, Volume 4, one would have read: “In Craven Street is a house, No  7, remarkable for having been the residence of Dr  Benjamin Franklin [14] In 1815, the identical phrase appeared in The Beauties of England and Wales [15] After 23 editions of not mentioning Franklin, his name finally appeared in the 24th edition of The Picture of London in 1826: “The house, No  7, Craven Street, in the Strand, was once the residence of Dr  Benjamin Franklin ”[16] In 1840, Jared Sparks referred to Franklin’s Craven Street residence appearing in London guide books in his voluminous The Works of Benjamin Franklin: “In the London Guide Books, ‘No  7, Craven Street,’ is still indicated as the house in which Dr  Franklin resided ”[17] In 1846, George Gulliver F R S , in his book, The Works of William Hewson, wrote: “She [Polly] had been upon terms of the warmest friendship with Dr  Franklin
Figure 5  No  7 Craven Street with Memorial Tablet  (Photo courtesy of British History Online, and the Survey of London)
since she was eighteen years of age  That eminent philosopher resided with her mother, Mrs  Margaret Stevenson, at No  7, Craven Street, Strand, during the fifteen years of his abode in London ”[18] Guide books mentioning Franklin at No  7 continued to proliferate throughout the century: Handbook for London; Past and Present, Volume I (1849);”[19] Handbook for Modern London (1851);”[20] The Town; Its Memorable Characters and Events (1859);”[21] London and Its Environs (1879) [22] There was an anomaly when London In 1880 Illustrated With Bird’s-Eye Views of the Principal Streets, Sixth Edition (1880) placed Franklin at 27 Craven street [23] The anomaly lasted for six years until his place of residence was changed to No  7 in the revised edition, London  Illustrated by Eighteen Bird’s-Eye Views of the Principal Streets (1886) [24] London Past and Present; Its History, Associations, and Traditions, Volume 1 (1891), copied the 1849 Handbook for London almost word-for-word and included, “The house is on the right from the Strand ”[25] In October of 1867, The Society of Arts in London declared that: “In order to show how rich the metropolis is in the memory of important personages and events, which it would be desirable to mark by means of tablets on houses, the Council have caused an alphabetical list to be prepared, … ”[26] Franklin had been elected a corresponding member to the Society in 1756 and was a popular choice among Council members deciding who they were to memorialize [27] By January of 1870, a tablet honoring him was affixed to the house they believed to have been his residence while in London, No  7 Craven Street in the Strand on the west side of the street [28] A majority of historians writing about Franklin in the nineteenth and early twentieth century placed him at No  7: O  L  Holley, The Life of Benjamin Franklin (1848); E  M  Tomkinson, Benjamin Franklin (1885); John Torrey Morse, Benjamin Franklin (1891); Paul Elmer More, Benjamin Franklin (1900); John S  C  Abbot, Benjamin Franklin (1903); Sydney George Fisher, The True Benjamin Franklin (1903)  A notable exception is D  H  Montgomery’s His Life Written by Himself published in 1896  He has Franklin at No  27 Craven Street  It seems then that depending upon the source, Franklin was thought to have lived at either No  7 or No  27, but not both, the overwhelming majority favoring No  7  As late as 2011, Franklin is still mentioned as living at No  7 [29]
In 1913, No  7 was scheduled to be torn down  An article in the March 1914 edition of The Book News Monthly, describes the situation:
As is well known to informed American pilgrims, it has been possible for all admirers of the famous philosopher and statesman to pay their respects to his memory before that house, No  7 Craven Street, just off the Strand, which was his chief home during his two sojourns in the British capital, but even as these lines are being written the London newspapers are recording that that interesting shrine is soon to be pulled down to make room for a restaurant  It is some mitigation of this misfortune to remember that at the most the Craven Street house was nothing more than a reproduction of the one in which Franklin had his suite of four rooms, for the structure has been rebuilt since Franklin’s time  When, then, some one makes a piteous plea that at least the philosopher’s bedroom shall be preserved, the soothing answer is that the apartment in question is only a replica of that in which the illustrious American enjoyed his well-earned slumbers in 1757-62 and 1764-75  The restaurant-builder, however, with an eye doubtless to possible American patronage, has assured the world that every effort will be made to preserve as much as possible of the entire structure [30]
Concerned with the possible demolition of Franklin’s residence, the Royal Society of Arts (formerly the Society of Arts[31]) initiated an inquiry into the matter [32] The London County Council, having taken over the responsibility of placing memorial tablets on notable houses from the Royal Society, was charged with the investigation  It ultimately fell to Sir George Laurence Gomme, a clerk to the Council, to come up with a response  A few years earlier Sir George had discovered Margaret Stevenson residing at No  27 Craven Street in the Westminster Rate Books  He must have wondered why No  7 on the west side of Craven Street was being celebrated as Franklin’s residence when the evidence clearly showed otherwise 
Sir George and his staff examined the various London directories discussed earlier and came up with a novel explanation for the discrepancy  They concluded that there had been two numbering systems on Craven Street  An anonymous author echoes Sir George’s conclusion about the two numbering systems in an article in The Journal of the Royal Society of Arts:
…an inspection of the directories of that time proves that there were at least two systems of numbering in Craven Street before the erection of the additional houses  According to one of these the numbers started from the top (Strand end) on the west side of the street, and ran down to the bottom to No  20, then crossed over and went back to the Strand along the east side – 21 to 36  According to the other system, the east side of the street was numbered from the bottom upwards, starting at No 1  This was not apparently in general use, but there is evidence that this numbering was at all events occasionally used 
The evidence of these two systems of numbering, and for believing that Mrs  Stevenson’s house was first No  7 under the oldest system, next No  27 under the second system, and finally No  36 under the latest and existing system, is to be found in the various directories and the Westminster rate-books [33]
The “evidence” mentioned above consisted of The London Directory’s listing of Brown & Whiteford at No  9: “The rate-books for 1781 and 1786 show the house next but one to the north of Mrs  Stevenson’s house as in the occupation of Brown and ‘Whiteford,’ while the old directories mention the business of the firm as wine merchants, and give their address as 9, Craven Street – then a little later, down to 1791, as 29, Craven Street  Curiously enough, in the years 1778 to 1780, or 1781, Lowndes gives it as No  9, and Kent as 29 ”[34] Ignoring Kent’s Directory having Brown and Whiteford as 29 and The London Directory (Lowndes) having Brown and Whiteford “a little later” as 29, and knowing that Mrs  Stevenson lived two doors south of them, Sir George concluded that her house must have been numbered 7, even though there is no listing in any of the directories of her residence ever being No  7  He surmised that the No  7 on the west side of Craven Street with the memorial tablet thought to have been Franklin’s residence had simply been confused with number 7 (27) on the east side  Again from The Journal of the Royal Society of Arts:
Taking all the evidence together, there cannot be any doubt whatever that Mrs  Stevenson’s house, in which Franklin lodged, was the house two doors north from Craven Court, first numbered 7, afterwards 27, and finally 36, and consequently that the house in which Franklin lived was that now numbered 36, not the one now numbered 7, on which the tablet is placed [35]
A response to The Royal Society of Arts was issued: “… the London County Council … informed the Society that it had made a mistake and that No  36 Craven street was the building that deserved commemoration ”[36] The Society accepted the Council’s conclusion, and despite assurances of preservation by the restaurant builder, No  7 was torn down the following year 
Sir George’s assertion “that Mrs  Stevenson’s house, in which Franklin lodged, was the house two doors north from Craven Court” was correct, however, his assertion that it was “first numbered 7, afterwards 27”, was not  It was only by association with the errant entry of Brown & Whiteford at No  9 from 1776-1782 in The London Directory that Mrs  Stevenson’s address was conjured to be No  7  The problem with associating her address exclusively with that of Brown & Whiteford at No  9 during those years is that, as previously demonstrated, The London Directory also listed four other Craven Street residents, Bond, Rowles, Sneyd, and Michie, who’s addresses did conform to the numbering system in The Westminster Rate Books  If Brown & Whiteford at No  9 was indicative of a numbering system different from The Westminster Rate Books, Bond, Rowles, Sneyd, and Michie would have been listed as Nos  10, 11, 12, and 15, respectively  So on one hand Sir George was relying on the Westminster Rate Books to establish Mrs  Stevenson at No  27 and on the other hand he was dismissing the Westminster Rate Books to establish her at No  7  Instead of using the anomalous listing of Brown & Whiteford at No  9, he could have just as easily, and more logically, used the Bond et al  listings, or the post-1782 Brown & Whiteford listing in the London Directory at No  29 to establish Mrs  Stevenson at No  27  Even if there had been two numbering systems, his assertion that No  27 was first numbered 7 would still be false  The earliest numbering system was the Westminster Rate Books dating from the early 1730s when the houses were constructed  Brown & Whiteford at No  9 didn’t appear until 46 years later and then only for a brief period 
There is ample evidence in Franklin’s correspondence and in a memoir by Polly Hewson (Mrs  Stevenson’s daughter) that Benjamin and Mrs  Stevenson lived in not one, but two houses on Craven Street  On July 6, 1772, Polly wrote to Benjamin from her house at Broad Street North in London: “My Mother I must tell you went off last friday week, took our little Boy with her and left Mr  Hewson [Polly’s husband, William] the care of her House [27 Craven Street]  The first thing he did was pulling down a part of it in order to turn it to his own purpose, and advantage we hope  This Demolition cannot affect you, who at present are not even a Lodger [Benjamin was traveling at the time], your litterary apartment remains untouch’d, the Door is lock’d …”[37] In a memoir about her husband written after his death Polly writes: “He [William Hewson] began his Lectures Sept  30, 1772, in Craven-street, where he had built a Theatre adjoining a house which he intended for the future residence of his family ”[38] On October 7, 1772, Benjamin wrote to his son William: “I am very well  But we [Mrs  Stevenson and I] are moving to another House in the same street; and I go down tomorrow to Lord LeDespencer’s to [stay a] Week till things are settled ”[39] To his son-in-law, Richard Bache, on the same day he wrote: “We are moving to another House in the [street] leaving this to Mr  Hewson ”[40] Writing to a friend on October 30, 1772 he explained: “I should sooner have answered your Questions but that in the Confusion of my Papers, occasioned by removing to another House, I could not readily find the Memorandums …”[41] On November 4, 1772 Benjamin informed his wife Deborah of the move  “We are removed to a more convenient House in the same street, Mrs  Stevenson having accommodated her Son-in-Law with that we lived in  The Removing has been a troublesome Affair, but is now over ”[42]
An agreement had been struck between the parties  Margaret and Benjamin would move to another house on Craven Street and allow Polly and William to move into No  27, the large yard behind the house being spacious enough to accommodate the anatomy school William wished to build [43] Perhaps the idea was inspired by Margaret’s next-door neighbor at No  26, Dr  John Leake, a man-midwife and founder of the Westminster Lying-in Hospital, who had built a theater adjoining his residence in which he practiced anatomy and taught midwifery [44]
After Margaret and Benjamin vacated No  27, Polly, William, their son William Jr , and William’s younger sister, Dorothy Hewson, took up residence there [45] In the 1773 Westminster Rate Books for Craven Street, Mrs  Stevenson’s (Stephenson in the Rate Books) name has been crossed out and replaced with “William Hewson ”[46] Further proof that the Hewsons had indeed moved into 27 Craven Street has been confirmed by the discovery of human and animal remains buried in the basement of No  36 (formerly No  27 and now the Benjamin Franklin House), a by-product of the dissections that took place at William’s anatomy school [47]
So what house on Craven Street did Mrs  Stevenson and Benjamin move into after vacating No  27? An examination of the Westminster Rate Books for the years 1774 and 1775 reveal them living not at No  7 on the west side of Craven Street as one might expect from the overwhelming consensus of nineteenth century guidebooks and biographies, but surprisingly at No  1 [48] The controversy of No  7 being torn down was all for naught as it had never been Franklin’s residence  Sir George was correct on that point  Unfortunately, No  1 was torn down as well in the early part of the twentieth century  The first time No  1 is mentioned as Franklin’s second residence is in the Survey of London: Volume 18, St Martin-in-The-Fields II: the Strand published by the London County Council in 1937, ironically the same County Council that had declared No  36 as Franklin’s only residence twenty-four years earlier 
From 1748 until 1772 Margaret ‘Stephenson’ occupied this house [No  27 (36)], and it was there that Benjamin Franklin settled after his arrival in London in 1757 as Agent to the General Assembly of Pennsylvania … In October, 1772, Mrs  Stevenson and Franklin removed to No  1, Craven Street (now demolished), and No  36 was for the next two years occupied by William Hewson, surgeon, who had married Mary Stevenson [49]
In the spring of 1774, William Hewson died unexpectedly of septicemia two weeks after cutting himself while dissecting a cadaver  Polly was left to care for their two young sons and was pregnant with a daughter she would give birth to in August of the same year  Is it possible that Margaret and Benjamin moved back into No  27 to assist Polly after the death of her husband as suggested in The Americanization of Benjamin Franklin?[50]
If the Westminster Rate Books are to be believed, the answer is no  For the year 1774, the Rate Books list Margaret Stevenson at No  1 and William Hewson at No  27  For the year 1775, they list Margaret Stevenson at No  1 and Magnus Falkner (Falconer/Falconar) at No  27  Magnus was William’s assistant at the anatomy school and fiancé to William’s sister, Dorothy  On his death bed, William instructed Polly, “let Mr  Falconar be my successor ”[51] Magnus would immediately take over the running of the anatomy school and continue William’s unfinished research  Four months later, he and Dorothy would marry [52] Essentially only two things changed at 27 Craven Street after William’s death: Polly gave birth to her daughter, and Magnus replaced William as the lease holder, so even if Margaret and Benjamin had wished to move back into No  27, there would have been no room for them  It is also interesting to note that considering the multiple times Benjamin wrote of his move out of No  27 (and complained of it), he never once mentioned moving back into No  27 in any of his correspondence after Mr  Hewson’s death 
Figure 6  No  36 Craven Street  (Photo courtesy of David Ross, britainexpress com)
In sum, based on the Westminster Rate Books[53] and Franklin’s correspondence, Mrs  Stevenson is known to have resided at No  27 (36) Craven Street from 1748 to 1772  It follows that, aside from the two years Franklin spent in Philadelphia from 1762 to 1764, he resided there from 1757 to 1772  Franklin’s correspondence also reveals that in the autumn of 1772, he and Mrs  Stevenson moved to another house on Craven Street  The 1773 Westminster Rate Books show her name crossed off at No  27 and William Hewson’s inserted  The following year the Rate Books list her at No  1 Craven Street  Evidence for Mrs  Stevenson and Benjamin remaining at No  1 after William’s death appears in the Westminster Rate Books for 1775 which have Mrs  Stevenson still residing at No  1 and Magnus Falkner residing at No  27  Further evidence can be construed from the lack of any mention of a move back into No  27 in Franklin’s correspondence  Despite the many theories one could devise as to why Franklin was thought to have lived at No  7 Craven Street by so many guide books and Franklin biographers of the nineteenth century, one thing is certain; at some point after Franklin’s departure to America in March of 1775, and no later than 1807, someone mistakenly associated him with No  7 on the west side of Craven Street, and it soon became his de facto residence  Credit must go to D  H  Montgomery in 1896 and Sir George in 1913 for setting the record partially straight by placing Franklin at No  27(36)  In 1937, the London County Council gave us the first accurate account of Franklin’s residences on Craven Street in the Survey of London at No  27(36) and No  1  It has been shown conclusively that No  27 was never previously numbered 7  It was, however, renumbered 36 in 1792 after ten additional houses were built at the southern end of the street and remains No  36 to this day 
[1] “Craven Street and Hungerford Lane”, in Survey of London: Volume 18, St Martin-in-the-Fields II: the Strand, ed  G H Gater and E P Wheeler (London, 1937), 27-39, Early History of the Site 
http://www british-history ac uk/survey-london/vol18/pt2/pp27-39
[2] “England, Westminster Rate Books, 1634-1900,” from database with images, Craven Street – 1735, FamilySearch from database by FindMyPast and images digitized by FamilySearch; citing Westminster City Archives, London 
[3] Ibid , Craven Street – 1748 
[4] The Statutes at Large, From Magna Charta to the End of the Eleventh Parliament of Great Britain  Anno 1761 Continued, Vol  XXVII, ed  Danby Pickering, (Cambridge, John Archdeacon, 1767), 96 
[6] James Raven, Publishing Business in Eighteenth-Century England, (Woodbridge: The Boydell Press, 2014), 201 
[7] The London Directory For the Year 1776, Ninth Edition, (London: T  Lowndes, 1776), title page 
[8] Kent’s Directory For the Year 1778, Forty-Sixth Edition, (London: Richard and Henry Causton, 1778), title page 
[9] A listing in Kent’s Directory for the Year 1882 on p  28 reveals, “Brown Sarah, Leather-seller, 1, Westmoreland-buildings, Aldersgate-street”, and in Kent’s Directory for the Year 1883 on p  175, “Whiteland Mary, Wine & Brandy Mercht  Jermyn-str  St  James ”
[10] “The Papers of Benjamin Franklin,” Sponsored by The American Philosophical Society and Yale University, Digital Edition by The Packard Humanities Institute, 22:263a 
http://franklinpapers org/franklin
Mrs  Stevenson wrote to Benjamin Franklin a letter from her new home at 75 Northumberland Court on November 16, 1775: “In this Court I have a kind friend, Mr  Lechmoen he comes and seats with me and talks of you with a hiy regard and friendship ”
[11] Survey of London, Early History of the Site 
[12] Survey of London, Footnotes/n 10 
[13] Survey of London, Historical Notes/No  31 
[14] David Hughson, LL D , London; Being An Accurate History And Description Of The British Metropolis And Its Neighbourhood, To Thirty Miles Extent, From An Actual Perambulation, Vol  IV, (London: W  Stratford, 1807), 227 
[15] The Reverend Joseph Nightingale, The Beauties of England and Wales: Or, Original Delineations, Topographical, Historical, and Descriptive, of Each County, Vol  X, Part III, Vol  II (London: J  Harris; Longman and Co ; J  Walker; R  Baldwin; Sherwood and Co ; J  and J  Cundee; B  and R  Crosby and Co ; J Cuthell; J  and J  Richardson; Cadell and Davies; C  and J  Rivington; and G  Cowie and Co , 1815), 245 
[16] John Britton, F S A  & Co , ed , The Original Picture of London, Enlarged and Improved: Being A Correct Guide For The Stranger, As Well As For the Inhabitant, To The Metropolis Of The British Empire Together With A Description Of The Environs, The Twenty-Fourth Edition (London: Longman, Rees, Orme, Brown, and Green, 1826), 479 
[17] Jared Sparks, The Works of Benjamin Franklin, Vol  VII, (Philadelphia: Childs & Peterson, 1840), 151 
[18] George Gulliver, F R S , The Works of William Hewson, F  R  S , (London: Printed for the Sydenham Society, MDCCCXLVI), xx 
[19] Peter Cunningham, Handbook for London; Past and Present, Vol  I, (London: John Murray, 1849), 245 
[20] F  Saunders, Memories of the Great Metropolis: or, London, from the Tower to the Crystal Palace, (New York: G P  Putnam, MDCCCLII), 138 
[21] Leigh Hunt, The Town; Its Memorable Characters and Events, (London: Smith, Elder and Co , 1859), 185 
[22] K  Baedeker, London and Its Environs, Including Excursions To Brighton, The Isle of Wight, Etc : Handbook For Travelers, Second Edition, (London: Dulau and Co , 1879), 133 
[23] Herbert Fry, London In 1880 Illustrated With Bird’s-Eye Views of the Principal Streets, Sixth Edition, (New York: Scribner, Welford, & Co , 1880), 50 
[24] Herbert Fry, London  Illustrated By Eighteen Bird’s-Eye Views of the Principal Streets, (London: W  H  Allen and Co , 1886), 40 
[25] Henry B  Wheatley, F S A , London Past and Present; Its History, Associations, and Traditions, Vol  1, (London: John Murray, New York: Scribner & Welford, 1891), 473 
[26] The Journal of the Society of Arts, Vol  XV, No  778, (October 18, 1867): 717 
[27] D  G  C  Allen, “Dear and Serviceable to Each Other: Benjamin Franklin and the Royal Society of Arts,” American Philosophical Society, Vol  144, No  3, (September 2000): 248-249 
Franklin was a corresponding member in 1756 because he was still residing in Philadelphia  He became an active member the following year when he moved to London 
[28] The Journal of the Society of Arts, Vol  XVIII, No  894, (Jan  7, 1870): 137 
“Since the last announcement, the following tablets have been affixed on houses formerly occupied by – Benjamin Franklin, 7 Craven-street, Strand, W C ”
[29] Franklin in His Own Time, eds  Kevin J  Haytes and Isabelle Bour, (Iowa City, University of Iowa Press, 2011), xxxvii 
“Takes lodgings with Margaret Stevenson at No  7 Craven Street ” It is unknown if the editors are referring to No  7 on the west side of Craven Street or No  36 on the east side using Sir George’s explanation of No  36 being previously numbered 7 
[30] Henry C  Shelly, “American Shrines on English Soil, III  In the Footprints of Benjamin Franklin,” in The Book News Monthly, September, 1913 to August, 1914, (Philadelphia: John Wanamaker, 1914), 325 
[31] The Journal of the Royal Society of Arts, Vol  LVI, No  2,880, (Jan  31, 1908): 245 
http://babel hathitrust org/cgi/pt?id=mdp 39015058423073;view=1up;seq=251
“His Majesty the King, who is Patron of the Society, has granted permission to the Society to prefix to its title the term ‘Royal,’ and the Society will consequently be known in future as the ‘Royal Society of Arts ’”
[32] Nineteenth Annual Report, 1914, of the American Scenic and Historic Preservation Society, (Albany: J  B  Lyon Company, 1914), 293 
http://babel hathitrust org/cgi/pt?id=wu 89072985302;view=1up;seq=4;size=150
[33] The Journal of the Society of Arts, Vol  LXII, No  3,183, (Nov  21, 1913): 18 
http://babel hathitrust org/cgi/pt?id=mdp 39015058422968;view=1up;seq=26
[36] Allen, “Dear and Serviceable,” 263-264 
[37] Papers of Benjamin Franklin, 19:20 
[38] Thomas Joseph Pettigrew, F  L  S , Memoirs of the Life and Writings of the Late John Coakley Lettsom With a Selection From His Correspondence, Vol  I, (London: Nichols, Son, and Bentley, 1817), 144 of Correspondence 
[39] Papers of Benjamin Franklin, 19:321b 
[40] Ibid , 19:314 
[41] Ibid , 19:353a 
[43] Simon David John Chaplin, John Hunter and the ‘museum oeconomy’, 1750-1800, Department of History, King’s College London  Thesis submitted for the degree of Doctor of Philosophy of the University of London , 202 
“Following Falconar’s death [1778] the lease [27 Craven Street] was advertised, and the buildings were described as:
A genteel and commodious house, in good Repair, with Coach-house and Stabling for two Horses…consisting of two rooms and light closets on each floor, with outbuildings in the Yard, a Museum, a Compleat Theatre, and other conveniences  (Daily Advertiser, 27 August 1778)”
[44] Simon Chaplin, “Dissection and Display in Eighteenth-Century London,” in Anatomical Dissection in Enlightenment England and Beyond: Autopsy, Pathology and Display, ed  Dr  Piers Mitchell, (Burlington: Ashgate Publishing Company, 2012), 108 
“Given that a nearby building at 35 [ No  26 in Franklin’s time] was occupied by the man-midwife John Leake, who advertised lectures – including lessons in the art of making preparations – at his ‘theatre’ between 1764 and 1788, it is possible that some facilities were shared  In both cases, however, the buildings [Leake’s residence at No  26 and Hewson’s residence next door at 27] served a dual function as domestic accommodation and as sites for lecturing and dissection ”
[45] George Gulliver, F R S , The Works of William Hewson, F  R  S , (London: Printed for the Sydenham Society, MDCCCXLVI), xviii 
[46] Westminster Rate Books, Craven Street – 1773, courtesy of the City of Westminster Archives 
[47] S W  Hillson et al , “Benjamin Franklin, William Hewson, and the Craven Street Bones,” Archaeology International, Vol  2, (Nov  22, 1998): 14-16 
http://dx doi org/10 5334/ai 0206
[48] Westminster Rate Books, Craven Street – 1774, 1775, courtesy of the City of Westminster Archives 
[49] Survey of London, Historical Notes/No  36, Craven Street (not sourced) 
[50] Gordon S  Wood, The Americanization of Benjamin Franklin, (New York: The Penguin Press, 2004), 261 
[51] Pettigrew, Memoirs, 146 of Correspondence 
[52] http://founders archives gov/documents/Franklin/01-22-02-0178, note 7  “Falconar married Hewson’s sister five months after the Doctor’s death; most of the Craven Street circle attended the wedding, and BF gave away the bride: Polly to Barbara Hewson, Oct  4, 1774, APS” (American Philosophical Society); “England Marriages, 1538–1973 ,” database, FamilySearch (https://familysearch org/ark:/61903/1:1:V52W-TGS : accessed September 15, 2015), Magnus Falconar and Dorothy Hewson, September 12, 1774; citing Saint Martin In The Fields, Westminster, London, England, reference ; FHL microfilm 561156, 561157, 561158, 942 B4HA V  25, 942 B4HA V  66 
[53] I chose to rely on the Westminster Rate Books for the numbering system on Craven Street  The books were consistent throughout the eighteenth century in the ordering of residents on the street and were used as the basis for the 1792 re-numbering  For the most part, commercial directories aligned with them as well  If by chance a directory didn’t initially align, it would inevitably produce future editions that did 
Benjamin Franklin, Benjamin Franklin House, London
More from David Turnquist
If one looked into Benjamin Franklin’s time on Craven Street, they might   
I think it’s very ironic that on the street maps included in your excellent article, Craven Street is so close to Scotland Yard  Because following the back and forth juxtapositions of numbers 7, 27 and 36 Craven Street (throw in 75 Northumberland Court and 1 Craven Street, too) was a case that could confound Sherlock Holmes 
Excellent job of deciphering street renumbering material spanning sixty years, including that of a wrong house number (# 7) being erroneously identified and then perpetuated in subsequent street map printings  It’s gratifying at least to know that the present day #36 Craven Street is the correct house for Ben Franklin tourists to visit  Except for #1 Craven Street for the last three years Franklin was in London, but we won’t get into that 
Again, excellent article, David!



ground_truth: Wearable sensors.

prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: What did the London Directory proclaim to contain?

Read the following text and answer briefly.

Context 4: 
(b) Agency Spotter may also offer the Services through applications built using Agency Spotter’s platform (“Agency Spotter Applications”), including smart phone applications, “Share” and other similar buttons and other interactive plugins distributed on websites across the Internet  Agency Spotter Applications are distinct from Third-Party Materials and applications address in Section 14(a), above  If you use an Agency Spotter application or interact with a website that has deployed a plugin, you agree that information about you and your use of the Services, including, but not limited to, your device, your mobile carrier, your internet access provider, your physical location, and/or web pages containing Agency Spotter plugins that load in your browser may be communicated to us  You acknowledge that you are responsible for all charges and necessary permissions related to accessing Agency Spotter through your mobile access provider  You should therefore check with your provider to find out if the Services are available and the terms for these services for your specific mobile devices  Finally, by using any downloadable application to enable your use of the Services, you are explicitly confirming your acceptance of the terms of the End User License Agreement associated with the application provided at download or installation, or as may be updated from time to time 
16  International Use  Agency Spotter makes no representation that materials on this site are appropriate or available for use in locations outside the United States, and accessing them from territories where their contents are illegal is prohibited  Those who choose to access this site from other locations do so on their own initiative and are responsible for compliance with local laws 
17  Dispute Resolution  These Terms and any claim, cause of action or dispute (“claim”) arising out of or related to these Terms shall be governed by the laws of the State of Georgia, regardless of your country of origin or where you access Agency Spotter, and notwithstanding any conflicts of law principles and the United Nations Convention for the International Sale of Goods  You and Agency Spotter agree that all claims arising out of or related to these Terms must be resolved exclusively by a state or federal court located in Fulton County, Georgia, except as otherwise mutually agreed in writing by the parties or as described in the Arbitration option in Section 16(b), below  You and Agency Spotter agree to submit to the personal jurisdiction of the courts located within Fulton County, Georgia, for the purpose of litigating all such claims  Notwithstanding the foregoing, you agree that Agency Spotter shall still be allowed to seek injunctive remedies (or an equivalent type of urgent legal relief) in any jurisdiction 
18  Arbitration  You agree that any dispute, claim or controversy arising hereunder or relating in any way to the Terms, shall be settled by binding arbitration in Fulton County, Georgia, in accordance with the commercial arbitration rules of Judicial Arbitration and Mediation Services (“JAMS”)  The arbitrator shall issue a written decision specifying the basis for the award made  The party filing a claim or counterclaim in the arbitration proceeding shall pay the deposit(s) determined by JAMS with respect to such claim or counterclaim  All other costs associated with the arbitration and imposed by JAMS shall be paid as determined by the arbitrator(s) and, in absence of such determination, equally by each party to the arbitration  In addition, unless the arbitrator awards payment of reasonable attorney and other fees to a party, each party to the arbitration shall be responsible for its own attorneys’ fees and other professional fees incurred in connection with the arbitration  Determinations of the arbitrator will be final and binding upon the parties to the arbitration, and judgment upon the award rendered by the arbitrator may be entered in any court having jurisdiction, or application may be made to such court for a judicial acceptance of the award and an order of enforcement, as the case may be  The arbitrator shall apply the substantive law of the State of Georgia, without giving effect to its conflict of laws rules 
19  Export Control  You agree to comply with all relevant export laws and regulations, including, but not limited to, the U S  Export Administration Regulations and Executive Orders (“Export Controls”)  You warrant that you are not a person, company or destination restricted or prohibited by Export Controls (“Restricted Person”)  You will not, directly or indirectly, export, re-export, divert, or transfer the Site or Service or any related software, any portion thereof or any materials, items or technology relating to Agency Spotter’s business or related technical data or any direct product thereof to any Restricted Person, or otherwise to any end user and without obtaining the required authorizations from the appropriate governmental entities 
(a) These Terms will continue until terminated in accordance with this Section 
(b) You may cancel your legal agreement with Agency Spotter at any time by (i) notifying Agency Spotter in writing, (ii) ceasing to use the Services, and (iii) closing your accounts for all of the Services which you use, if we have made this option available to you  Your cancellation of the Services will not alter your obligation to pay all charges incurred prior to your effective date of termination 
Agency Spotter may terminate its legal agreement with you if, (i) you have breached any provision of the Terms (or have acted in manner which clearly shows that you do not intend to, or are unable to comply with the provisions of the Terms), or (ii) Agency Spotter is required to do so by law (for example, where the provision of the Services to you is, or becomes, unlawful), or (iii) Agency Spotter is transitioning to no longer providing the Services to users in the country in which you are resident or from which you use the service, or (iv) the provision of the Services to you by Agency Spotter is, in Agency Spotters’ opinion, no longer commercially viable 
(c) The terms provided in Sections 2, 3, 6, 11, 12, 13, 14, 17, 19, 20, 21 and 22 of these Terms shall survive any termination of these Terms 
21  Independent Contractors  The parties are and intend to be independent contractors with respect to the Services contemplated hereunder  You agree that neither you nor any of your employees or contractors shall be considered as having an employee status with Agency Spotter  No form of joint employer, joint venture, partnership, or similar relationship between the parties is intended or hereby created 
22  Assignment and Delegation  You may not assign or delegate any rights or obligations under these Terms  Any purported assignment or delegation shall be ineffective  We may freely assign or delegate all rights and obligations under these Terms, fully or partially, without notice to you  We may also substitute, by way of unilateral novation, effective upon notice to you, Agency Spotter Inc  for any third party that assumes our rights and obligations under these Terms 
The personally identifiable information we collect from you allows us to provide you with the Services and to enable users to navigate and enjoy using the Site  We will also use your personally identifiable information to develop, improve and advertise the Site and Services  We may also use your personally identifiable information for internal purposes such as auditing, data analysis and research to improve our Services and customer communications  We do not rent, sell or otherwise provide your personally identifiable information to third parties without your consent, except as described in this policy or as required by law 
When you register with us through the Site or Services and become a Registered User, or when you wish to contact another Registered User, we will ask you for personally identifiable information  This refers to information about you that can be used to contact or identify you (“Personally Identifiable Information“)  Personally Identifiable Information includes, but is not limited to, your name, phone numbers, email address, home postal address, business address, social media user names, employer/affiliated organization, reasons for accessing the Site, and intended usage of requested information, but does not include your credit card number or billing information  We may also use your email address or phone number (if provided by you) to contact you regarding changes to the Services; system maintenance and outage issues; account issues; or otherwise to troubleshoot problems  In order to process some of your transactions through the Site and Services, we may also ask for your credit card number and other billing information (“Billing Information“; and, together with Personally Identifiable Information, “Personal Information“) 
Information you provide to us also includes your account profile and your contributions to discussion groups and community features Agency Spotter may offer  Do not upload or insert any information to or into the Site or Services that you do not want to be shared or used in the manner described in this section 
In addition, when you use the Site, our servers automatically record certain information that your web browser sends whenever you visit any website  These server logs may include information such as your web request, Internet Protocol address, browser type, browser language, referring/exit pages and URLs, platform type, number of clicks, domain names, landing pages, pages viewed and the order of those pages, the amount of time spent on particular pages, the date and time of your request, and one or more cookies that may uniquely identify your browser 
Information from third party services and other websites 
Do not upload or insert any information to or into the Site or Services that you do not want to be shared or used in the manner described in this section 
Advertisements  Advertisers who present ads on the Site may use technological methods to measure the effectiveness of their ads and to personalize advertising content  You may use your browser cookie settings to limit or prevent the placement of cookies by advertising networks  Agency Spotter does not share personally identifiable information with advertisers unless we get your permission 
Links  When you click on links on Agency Spotter you may leave our site  We are not responsible for the privacy practices of other sites, and we encourage you to read their privacy statements 
If we are requested to disclose your information to a government agency or official, we will do so if we believe in good faith, after considering your privacy interests and other relevant factors, that such disclosure is necessary to: (i) conform to legal requirements or comply with a legal process with which we are involved; (ii) protect our rights or property or the rights or property of our affiliated companies; (iii) prevent a crime or protect national security; or (iv) protect the personal safety of Site users or the public  Because Agency Spotter is a United States limited liability company and information collected on our Site is stored in whole or in part in the United States, your information may be subject to U S  law 
We also reserve the right to disclose Personally Identifiable Information and/or other information about users that Agency Spotter believes, in good faith, is appropriate or necessary to enforce our agreements, take precautions against liability, investigate and defend itself against ay third-party claims or allegations, assist government enforcement agencies, protect the security or integrity of our Site or Services, and protect the rights, property or personal safety of Agency Spotter, our users and others 
Cookies allow us to (i) manage, present and keep track of temporary information, such as data you upload onto the Site for use with the Services; (ii) register you as a Registered User on the Site or in other various programs associated with the Site; (iii) remember you when you log in to the places on the Site that require you to be a Registered User of the Site; (iv) help us understand the size of our audience and traffic patterns; (v) collect and record information about what you viewed on the Site; and (vi) deliver specific information to you based on your interests 
When you access the Site, the Site automatically collects certain non-personally identifiable information through the use of electronic images known as web beacons (sometimes called single-pixel gifs) and log files  Such information may include your IP address, browser type, the date, time and duration of your access and usage of the Site and whether you opened emails You received from us 
This information is collected for all visits to the Site and then analyzed in the aggregate  This information is useful for, among other things, tracking the performance of our online advertising, such as online banner ads, and determining where to place future advertising on other websites 
Editing your profile  You may review and change or remove your personal information or the settings for your Agency Spotter account at any time by going to your account profile  You can edit your name, email address, password and other account information here  Please be aware that even after your request for a change is processed, Agency Spotter may, for a time, retain residual information about you in its backup and/or archival copies of its database 
Deactivating or deleting your account  If you want to stop using your account you may deactivate it or delete it  When you deactivate an account, no user will be able to see it, but it will not be deleted  We save your profile information in case you later decide to reactivate your account  Many users deactivate their accounts for temporary reasons and in doing so are asking us to maintain their information until they return to Agency Spotter  You will still have the ability to reactivate your account and restore your profile in its entirety  When you delete an account, it is permanently deleted from Agency Spotter  You should only delete your account if you are certain you never want to reactivate it  You may deactivate your account or delete your account within your account profile 
Limitations on removal  Even after you remove information from your profile or delete your account, copies of that information may remain viewable elsewhere to the extent it has been shared with others, it was otherwise distributed pursuant to your privacy settings, or it was copied or stored by other users  However, your name will no longer be associated with that information on Agency Spotter  (For example, if you post something to another user’s or Agency’s profile or Agency’s portfolio and then you delete your account, that post may remain, but be attributed to an “Anonymous Agency Spotter User ”) Additionally, we may retain certain information to prevent identity theft and other misconduct even if deletion has been requested  If you have given third party applications or websites access to your information, they may retain your information to the extent permitted under their terms of service or privacy policies  But they will no longer be able to access the information through our platform after you disconnect from them 
Default Settings  Because the mission of Agency Spotter is to connect businesses and agencies, enabling them to save time, be more productive and successful, we have established what we believe are reasonable default settings that we have found most agencies and professionals desire  Because Registered Users may use and interact with Agency Spotter in a variety of ways, and because those uses may change over time, we designed our settings to provide our users control over the information they share  We encourage our Registered Users to review their account settings and adjust them in accordance with their preferences 
Risks inherent in sharing information  Please be aware that no security measures are perfect or impenetrable, and no method of transmission over the Internet, or method of electronic storage, is 100% secure  We cannot control the actions of other users with whom you share your information  We cannot guarantee that only authorized persons will view your information  We cannot ensure that information you share on the Site or through the Services will not become publicly available  We are not responsible for third party circumvention of any privacy or security measures on Agency Spotter  You can reduce these risks by using common sense security practices such as choosing a strong password, using different passwords for different services, and using up to date antivirus software 
If you receive an unsolicited email that appears to be from us or one of our members that requests personal information (such as your credit card, login, or password), or that asks you to verify or confirm your account or other personal information by clicking on a link, that email was likely to have been sent by someone trying to unlawfully obtain your information, sometimes referred to as a “phisher” or “spoofer ” We do not ask for this type of information in an email  Do not provide the information or click on the link  Please contact us at [email protected] if you get an email like this  Notwithstanding the foregoing, after your initial account setup, we may send an email to your registered account address solely to confirm that we have the correct, valid email address for your account 
If You have concerns about your privacy in connection with your use of the Site or any general questions related thereto, please tell us by emailing us at [email protected] We will make every reasonable effort to address your concerns 
Thank You for supporting websites, such as ours  We take your privacy seriously by implementing written privacy policies, such as this one 



ground_truth: An alphabetical list of names and places of abode of the merchants and principal traders of the cities of London and Westminster, the Borough of Southwark, and their environs, with the number affixed to each house.