Package: raman-hypmap
raman_hypmapRaman hyperspectral processing via python scripts tailoured for .txt output of Horiba instruments, e.g., theLabSpec 6 Spectroscopy Suite Softwareproduct. Theoretically this code will work on any hyperspectral Raman data if the .txt input is formatted correctly.Three python modules are available:bgr_process — automated background reduction of entire mapfilter_window — using defined peaks to create spectral intensity mapshypmap_plot — interactive plot of intensity maps to create and save spectral masksReferencingIf using this code, please reference the archive atzenodo.orgPlease reference Wang and Dai (2016) for the iterative polynomial smoothing method used for background reduction of spectrabgr_process. Reference: Wang, T. and Dai, L., 2017. Background subtraction of Raman spectra based on iterative polynomial smoothing. Applied spectroscopy, 71(6), pp.1169-1179. doi:https://doi.org/10.1177/0003702816670915.The progress bar used in this code is fromuserand full details are onstackoverflowThe pixel selector used for the hyperspectral maphypmap_plotis adapted from amatplotlib tutorialfor creating a polygon selector.LicenseThe project is licensed under the GNU-v3 license.InstallationTo install the package, a.tomland.whlare provided. Download, navigate to the folder where the ..tomlor.whlfile is located and runpip install. Alternatively, the files can be imported as modules locally. Module are defined withinhyprocess.py.This package is also available athttps://pypi.org/project/raman-hypmap/UsageImport a .txt hyperspectral map file from the horiba software (e.g.,LabSpec 6). The file is structured as a tab deliminated file with the first row denoting theRaman Shift(cm-1) values. Note this row is offset two spaces, given that each hyperspectral pixel is then stored asx-value,y-value, andspectral intensity-valueswhere the latter has the same length as the list ofRaman Shiftvalues, created pairs. An exampe .txt fie is provided inexample/example_data.txtto illustrate this.The .txt file should be imported into a pandas dataframe.Although optional, setting the coordinates of the map to start at (0,0) will avoid complications.import pandas as pd

file = 'example/example_data.txt'

file = file.rsplit('.',1)[0]
_df = pd.read_csv(file+'.txt', sep='\t', index_col=False, decimal=',',header=None)
_df = _df.replace(',','', regex=True)
# set origin to zero
_df[0] = _df[0].sub(_df[0].min())
_df[1] = _df[1].sub(_df[1].min())The python modules can then be run in order, for example:from raman_hypmap import hyprocess

''' --- background reduction --- '''
_df_bgr = hyprocess.bgr_process(_df,file)
_df_bgr = pd.read_csv(file+'_bgr.csv', sep=',', index_col=False,header=None)
_df_raw = pd.read_csv(file+'_raw.csv', sep=',', index_col=False,header=None)

''' --- mineral filter --- '''
window_location = 'example/_window_ranges.csv'
hyprocess.filter_window(_df_bgr,file+'_bgr',window_location)

''' --- interactive hyperspectral map --- '''
_df_filtered = pd.read_csv(file+'_bgr_w_filter.csv', sep=',', index_col=False)
hyprocess.hypmap_plot(_df_filtered,_df_raw,file)Note that each module imports the generated .csv file from the previous module as an input. These files will be created automatically when running the module to the local folder.An example script showing this code is also available in:example/example.pybgr_processThe input format is the standard .txt file generated by the HoribaLabSpec 6software. However, this format can be recreated via a tab-delimited .txt file as shown below:blankblankRaman-shift values (tab-delimited list)x-coorindatey-coordinateRaman-intensity values (tab-delimited list).........Each spectra is background reduced via an iterative polynomial smoothing method Wang and Dai (2016) and written into a pandas dataframe. Three files are output and automatically saved as .csv files:example_data_raw.csv — reformatted data input saved as a .csv fileexample_data_bg.csv — the background subtracted from each spectra saved as a spectral fileexample_data_bgr.csv — the resulting background reduced spectra saved as a spectral fileEach file is saved in a identical structure to the table shown above, but comma-delimited in a format that can be opened and viewed in Microsoft Excel.filter_windowThis module needs a .csv file formatted as explained in the section above to operate correctly. The code will itterate over every spectra provided and take the maximum intensity value from this window. It will then output a hyperspectral grid of values using this window. That hyperspecral grid will be saved as an image .png file with greyscale values stretched to the maximum and minimum intensity values recorded.The filter windows are imported from a standalone .csv file called_window_ranges.csvthat can be open and edited in Microsoft Excel. The headings must not be changed. The format of the spredsheet is shown below:phasechar_minchar_maxcomp_mincomp_maxi_mini_maxgraphite151016200002000apatite9509750002000.....................These numbers are important as they define how the python code operates:'Phase' defines the name of the species'char_min' and 'char_max' define the window that the filter will operate over. A new hyperspectral map will be created for every row of values.Sometimes spectra overlap and so 'comp_min' and 'comp_max' define a comparison window. the maximum intensity values defined in the 'char_' window must belargerthan the maximum intensity value in the 'comp_' window otherwise a value of 0 will be given for that spectra.'i_min' and 'i_max' work in a similar way to the above. The maximum intensity of the spectra in the window defined by 'char_' must be within the intensity thresholds defined by the 'i_min' and 'i_max' variables.Each row of values will create and save a new hyperspectral map using that window. Examples derived fromexample/example.pyusing the current configuration of_window_ranges.csvare shown below:In addition, this module will create a new_w_filter.csvspreadsheet that stores these values. This is formatted so that the first two columns store the x and y coordinates and each subsequent column is added for every species  defined in_window_ranges.csv. An example of the output is providedexample/example_output/example_data_bgr_w_filter.csvhypmap_plotThis module needs two .csv files, a_w_filter.csvspreadsheet output generated by thefilter_windowmodule and a spectrum data file formatted using thebgr_processmodule. The module will then create a hyperspectral image with a superposed grid of points aligned with each pixel in the hyperspectral image. These points can be selected by drawing circles using the mouse. When highlighted, they will be black.When pixels are selected in this way, pressingENTERwill generate a simple plot generated from every single pixel. Three lines are shown on the plot:Themeanwith ± 2 standard deviations with grey barsThemeansmoothed using a Savitzky–Golay filter (2nd order polynomial and 15-wide window).Themedianof all selected pixels (Orange)The location of the highlighted pixels will be saved as a .png file and the spectral mask will be saved as a tab-delimited .txt file. In addition, the .txt file saves thesumof all spectra. A second text file consisiting of two columns: (1) Raman shift values, vs (2) mean intensity will also be generated. This is a suitable input for additional automatic fitting software, such asIFORS. An example of these .txt files are providedexample/example_output/example_data_spectra.txt.This process can be repeated as long as a new selection is made. Every time you hitENTERand plot the result of a new mask, the files automatically saving the spectra and location will be overwritten.