Package: pyspectra
PyspectraWelcome to pyspectra.This package is intended to  put functions together to analyze and transform spectral data from multiple spectroscopy instruments.Currently supported input files are:.spc.dxPySpectra is intended to facilitate working with spectroscopy files in python by using a friendly  integration with pandas dataframe objects..
Also pyspectra provides a set of routines to execute spectral pre-processing like:MSCSNVDetrendSavitzky - GolayDerivatives..Data spectra can be used for traditional chemometrics analysis but also can be used in general advanced analytics modelling in order to deliver additional  information to manufacturing models by supplying spectral information.#Import basic librariesimportspcimportmatplotlib.pyplotaspltimportnumpyasnpimportpandasaspdfromsklearn.decompositionimportPCARead .spc fileRead a single filefrompyspectra.readers.read_spcimportread_spcspc=read_spc('pyspectra/sample_spectra/VIAVI/JDSU_Phar_Rotate_S06_1_20171009_1540.spc')spc.plot()plt.xlabel("nm")plt.ylabel("Abs")plt.grid(True)print(spc.head())gx-y(1)
908.100000    0.123968
914.294355    0.118613
920.488710    0.113342
926.683065    0.108641
932.877419    0.098678
dtype: float64Read multiple .spc files from a directoryfrompyspectra.readers.read_spcimportread_spc_dirdf_spc,dict_spc=read_spc_dir('pyspectra/sample_spectra/VIAVI')display(df_spc.transpose())f,ax=plt.subplots(1,figsize=(18,8))ax.plot(df_spc.transpose())plt.xlabel("nm")plt.ylabel("Abs")ax.legend(labels=list(df_spc.transpose().columns))plt.show()gx-y(1)
gx-y(1)
gx-y(1)
gx-y(1)
gx-y(1)
gx-y(1)
gx-y(1)
gx-y(1)<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>JDSU_Phar_Rotate_S06_1_20171009_1540.spcJDSU_Phar_Rotate_S11_2_20171009_1614.spcJDSU_Phar_Rotate_S17_1_20171009_1652.spcJDSU_Phar_Rotate_S23_1_20171009_1734.spcJDSU_Phar_Rotate_S30_2_20171009_1815.spcJDSU_Phar_Rotate_S37_2_20171009_1853.spcJDSU_Phar_Rotate_S43_2_20171009_1928.spcJDSU_Phar_Rotate_S49_1_20171009_2000.spc908.1000000.1239680.1647500.1566470.1478280.1828330.1719570.1644710.149373914.2943550.1186130.1599800.1507460.1429740.1784520.1668270.1595450.142818920.4887100.1133420.1551930.1449590.1381780.1737340.1616950.1543300.136648926.6830650.1086410.1513980.1401780.1340140.1700610.1571100.1498760.130452932.8774190.0986780.1418590.1297150.1244260.1605900.1470760.1401190.119561...........................1651.4225810.2209350.2620700.2596430.2429160.2790410.2714920.2606640.2527041657.6169350.2218480.2627320.2606640.2430920.2789620.2728930.2616470.2544811663.8112900.2199040.2603350.2589750.2406560.2763820.2716240.2602780.2537611670.0056450.2140800.2534750.2531100.2340470.2695280.2656150.2545680.2482881676.2000000.2042170.2423750.2430820.2235390.2587710.2553060.2448260.238663125 rows Ã— 8 columnsRead .dx spectral filesPyspectra is also built with a set of regex that allows to read the most common .dx file formats from different vendors like:FOSSSi-Ware SystemsSpectral EnginesTexas InstrumentsVIAVIRead a single .dx file.dx reader can read:Single files containing single spectra : readSingle files containing multiple spectra : readMultiple files from directory : read_from_dirSingle file, single spectra# Single file with single spectrafrompyspectra.readers.read_dximportread_dx#Instantiate an objectFoss_single=read_dx()# Run  read methoddf=Foss_single.read(file='pyspectra/sample_spectra/DX multiple files/Example1.dx')df.transpose().plot()<matplotlib.axes._subplots.AxesSubplot at 0x1f44faa7940>Single file, multiple spectra:.dx reader stores all the information as attributes of the object on Samples. Each key represent a sample.Foss_single=read_dx()# Run  read methoddf=Foss_single.read(file='pyspectra/sample_spectra/FOSS/FOSS.dx')df.transpose().plot(legend=False)<matplotlib.axes._subplots.AxesSubplot at 0x1f44f7f2e50>forcinFoss_single.Samples['29179'].keys():print(c)y
Conc
TITLE
JCAMP_DX
DATA TYPE
CLASS
DATE
DATA PROCESSING
XUNITS
YUNITS
XFACTOR
YFACTOR
FIRSTX
LASTX
MINY
MAXY
NPOINTS
FIRSTY
CONCENTRATIONS
XYDATA
X
YSpectra preprocessingPyspectra has a set of built in classes to perform spectra  pre-processing like:MSC: Multiplicative scattering correctionSNV: Standard normal variateDetrendn order derivativeSavitzky golay smmothingfrompyspectra.transformers.spectral_correctionimportmsc,detrend,sav_gol,snvMSC=msc()MSC.fit(df)df_msc=MSC.transform(df)f,ax=plt.subplots(2,1,figsize=(14,8))ax[0].plot(df.transpose())ax[0].set_title("Raw spectra")ax[1].plot(df_msc.transpose())ax[1].set_title("MSC spectra")plt.show()SNV=snv()df_snv=SNV.fit_transform(df)Detr=detrend()df_detrend=Detr.fit_transform(spc=df_snv,wave=np.array(df_snv.columns))f,ax=plt.subplots(3,1,figsize=(18,8))ax[0].plot(df.transpose())ax[0].set_title("Raw spectra")ax[1].plot(df_snv.transpose())ax[1].set_title("SNV spectra")ax[2].plot(df_detrend.transpose())ax[2].set_title("SNV+ Detrend spectra")plt.tight_layout()plt.show()Modelling of spectraDecompose using PCApca=PCA()pca.fit(df_msc)plt.figure(figsize=(18,8))plt.plot(range(1,len(pca.explained_variance_)+1),100*pca.explained_variance_.cumsum()/pca.explained_variance_.sum())plt.grid(True)plt.xlabel("Number of components")plt.ylabel(" cumulative% of explained variance")df_pca=pd.DataFrame(pca.transform(df_msc))plt.figure(figsize=(18,8))plt.plot(df_pca.loc[:,0:25].transpose())plt.title("Transformed spectra PCA")plt.ylabel("Response feature")plt.xlabel("Principal component")plt.grid(True)plt.show()Using automl libraries to deploy faster modelsimporttpotfromtpotimportTPOTRegressorfromsklearn.model_selectionimportRepeatedKFoldcv=RepeatedKFold(n_splits=10,n_repeats=3,random_state=1)model=TPOTRegressor(generations=10,population_size=50,scoring='neg_mean_absolute_error',cv=cv,verbosity=2,random_state=1,n_jobs=-1)y=Foss_single.Conc[:,0]x=df_pca.loc[:,0:25]model.fit(x,y)HBox(children=(FloatProgress(value=0.0, description='Optimization Progress', max=550.0, style=ProgressStyle(deâ€¦



Generation 1 - Current best internal CV score: -0.30965836730187607

Generation 2 - Current best internal CV score: -0.30965836730187607

Generation 3 - Current best internal CV score: -0.30965836730187607

Generation 4 - Current best internal CV score: -0.308295313408046

Generation 5 - Current best internal CV score: -0.308295313408046

Generation 6 - Current best internal CV score: -0.308295313408046

Generation 7 - Current best internal CV score: -0.308295313408046

Generation 8 - Current best internal CV score: -0.3082953134080456

Generation 9 - Current best internal CV score: -0.3082953134080456

Generation 10 - Current best internal CV score: -0.3078569602146527

Best pipeline: LassoLarsCV(PCA(LinearSVR(input_matrix, C=0.1, dual=True, epsilon=0.1, loss=epsilon_insensitive, tol=0.01), iterated_power=3, svd_solver=randomized), normalize=False)





TPOTRegressor(cv=RepeatedKFold(n_repeats=3, n_splits=10, random_state=1),
              generations=10, n_jobs=-1, population_size=50, random_state=1,
              scoring='neg_mean_absolute_error', verbosity=2)fromsklearn.metricsimportr2_scorer2=round(r2_score(y,model.predict(x)),2)plt.scatter(y,model.predict(x),alpha=0.5,color='r')plt.plot([y.min(),y.max()],[y.min(),y.max()],LineStyle='--',color='black')plt.xlabel("y actual")plt.ylabel("y predicted")plt.title("Spectra model prediction R^2:"+str(r2))plt.show()