DOI: 10.5061/dryad.bcc2fqzpt
title: In-sensor multilevel image adjustment for high-clarity contour extraction using adjustable synaptic phototransistors

description: Robotic vision has traditionally relied on high-performance yet
 resource-intensive computing solutions, which necessitate
 high-throughput data transmission from vision sensors to
 remote computing servers, sacrificing energy-efficiency and
 processing speed. A promising solution is data compaction through
 contour extraction, visualizing only the outlines of objects while
 eliminating superfluous backgrounds. Here, we introduce an
 in-sensor multi-level image adjustment method using adjustable
 synaptic phototransistors, enabling the capture of well-defined images
 with optimal brightness and contrast suitable for achieving
 high-clarity contour extraction. This is enabled by emulating
 dopamine-mediated neuronal excitability regulation
 mechanisms. Electrostatic gating effect either facilitates or
 inhibits time-dependent photocurrent accumulation, adjusting
 photo-responses to varying lighting conditions. Through excitatory and
 inhibitory modes, the adjustable synaptic phototransistor
 enhances visibility of dim and bright regions,
 respectively, facilitating distinct contour extraction and
 high-accuracy semantic segmentation. Evaluations using road
 images demonstrate improvement of both object detection accuracy and
 Intersection over Union, and significant compression of data
 volume.
description: # In-sensor multilevel image adjustment for high-clarity contour
 extraction using adjustable synaptic phototransistors
 [https://doi.org/10.5061/dryad.bcc2fqzpt](https://doi.org/10.5061/dryad.bcc2fqzpt) ## Description: This dataset supports the study titled "In-sensor multilevel image adjustment for high-clarity contour extraction using adjustable synaptic phototransistors." It includes Python scripts and associated data for preprocessing and analyzing image data using the parameter of synaptic phototransistor and in-sensor multilevel image adjustment method. ## **Directory and File Structure** * Adaptive_thresholding.py * Merge_using_logic_operation.py * Nonlinear_optoelectronic_conversion.py * Sobel_edge_detection.py * synaptic_phototransistor_parameter_for_nonlinear_optoelectronic_conversion.xlsx * \[raw_csv_data_folder] Folder with raw CSV image data (not provided here) * \[output_folders] Output folders for processed images (not provided here) ### **Processing Pipeline and Scripts** #### 0. Synaptic\_phototransistor\_parameterNonlinear Optoelectronic Conversion * **File: synaptic_phototransistor_parameter_for_nonlinear_optoelectronic_conversion.xlsx** - Function: This file provides the experimentally measured nonlinear optoelectronic response characteristics of the adjustable synaptic phototransistor under a 250 ms optical input condition. It is essential for device-inspired nonlinear preprocessing of raw image data. - Input: * Intensity Level (0-255). The raw grayscale pixel intensity value before conversion. This corresponds to the input light intensity to the phototransistor. - Output: * Response Value. The corresponding output electrical response of the synaptic phototransistor to each input intensity level and *V*el. This value was experimentally obtained and represents the nonlinear behavior of the device. These output response values are directly utilized to construct phototransistor_parameter_250ms.csv used in Nonlinear_optoelectronic_conversion.py. This enables mapping raw pixel intensities to the actual electrical characteristics of the phototransistor, reflecting its nonlinear optoelectronic properties. #### 1. Nonlinear Optoelectronic Conversion * **File: Nonlinear_optoelectronic_conversion.py** * Function: Converts raw pixel data from CSV files using intensity-dependent synaptic phototransistor parameters. Adjusts data for nonlinear optoelectronic conversion using intensity-dependent parameters and normalizes the output for further processing. * Input: * CSV files containing raw grayscale image data. * Parameter file: synaptic_phototransistor_parameter_for_nonlinear_optoelectronic_conversion.xlsx * Output: * Processed CSV files saved in the same input folder. #### 2. Contour Extraction with Sobel Edge Detection * **File: Sobel_edge_detection.py** * Function: Extracts contours from processed CSV data through gradient computation using the Sobel operator, facilitating the evaluation of in-sensor multilevel image adjustment performance. * Input: Processed CSV files. * Output: PNG images of detected contours. #### 3. Contour Extraction with Adaptive Thresholding * **File: Adaptive_thresholding.py** * Function: Extracts contours from processed CSV data by applying adaptive thresholding for binary conversion, enabling the evaluation of in-sensor multilevel image adjustment performance. * Use Case: Effective for images with uneven lighting. * Input: Processed CSV files. * Output: PNG images of extracted contours. #### 4. Merging Contours Using Logical Operations * **File: Merge_using_logic_operation.py** * Function: Merges contour images (e.g., from excitatory/inhibitory modes) using logical AND/OR operations. * Input: * Two sets of processed contour data (from separate modes). * Output: Merged CSV files with refined contour information. ### **Example Usage Workflow** 1\. Preprocess:    Run Nonlinear_optoelectronic_conversion.py with input CSV folder 2\. Contour extraction (choose one method):    - Run Sobel_edge_detection.py    OR    - Run Adaptive_thresholding.py 3\. Merge results:    Run Merge_using_logic_operation.py to merge two contour images respectively obtained from the excitatory and inhibitory modes of the multilevel image adjustment process, using logical operations (AND/OR), to generate a refined contour image. ### **Variables Description** 1\. Nonlinear_optoelectronic_conversion.py | Variable | Description | | :------------------ | :---------------------------------------------------------------------------------------------------------------- | | csv\_folder\_path | Directory path where raw CSV image files are stored. Each file contains grayscale pixel intensity values (0-255). | | conversion\_path | File path of synaptic phototransistor parameter CSV file. | | conversion\_data | Data loaded from conversion\_path containing phototransistor response parameters. | | conversion\_max | Maximum output value from conversion\_data used for normalization. | | post\_optical\_data | Converted and normalized 2D pixel data array (values 0-255). | 2\. Sobel_edge_detection.py | Variable | Description | | :------------------- | :--------------------------------------------------------------------------------- | | csv\_folder\_path | Directory path of processed CSV image files (output of nonlinear conversion). | | output\_folder\_path | Directory path where edge-detected PNG images are saved. | | image | 2D numpy array loaded from CSV representing grayscale image. | | sobelX, sobelY | Gradient values in x and y directions calculated by Sobel operator. | | sobel | Gradient magnitude calculated from sobelX and sobelY, representing edge intensity. | 3\. Adaptive_thresholding.py | Variable | Description | | :------------------- | :---------------------------------------------------------------------------- | | csv\_folder\_path | Directory path of processed CSV image files (output of nonlinear conversion). | | output\_folder\_path | Directory path where binary contour PNG images are saved. | | image | 2D numpy array loaded from CSV representing grayscale image. | | adaptive\_threshold | Binary image obtained using adaptive thresholding (pixel values: 0 or 255). | 4\. Merge_using_logic_operation.py | Variable | Description | | :------------------- | :------------------------------------------------------------------------------------ | | folder1\_path | Directory path of contour images from excitatory mode (processed CSV). | | folder2\_path | Directory path of contour images from inhibitory mode (processed CSV). | | output\_folder\_path | Directory path where merged CSV files are saved. | | data1, data2 | 2D arrays loaded from folder1\_path and folder2\_path for merging. | | result | Merged 2D array obtained using logical AND (or OR) operation between data1 and data2. | Citation is encouraged when using this dataset. Please refer to the provided DOI. ### **Notes** * Input CSV files must be formatted with grayscale pixel intensity values (0–255). * The directories [raw_csv_data_folder] and [output_folders] are example placeholders. Users should create these folders as needed and place their own CSV-formatted grayscale image data in the input folder to run the scripts. * File paths must be set manually within each script. * Logical operation in the merge script (AND, OR) can be configured based on analysis needs.