Package: pynir
Commom Calibration methods for multivariate calibrationThis is a Python library for dealing with multivariate calibration, e.g., Near infrared spectra regression and classification tasks.InstallationUse the package managerpipto installpynir.pipinstallpynirIn addition, we have also provide an online version atlinkUsageSimulata NIR spectra (spc) and reference values (conc)frompynir.utilsimportsimulateNIRspc,conc=simulateNIR()Regressionfrompynir.utilsimportsimulateNIRfrompynir.Calibrationimportpls# estabilish PLS modeln_components=10plsModel=pls(n_components=n_components)plsModel.fit(X,y)yhat=plsModel.predict(X)Classification# simulate NIR datafrompynir.utilsimportsimulateNIRfrompynir.Calibrationimportplsdanclass=4X,y,wv=simulateNIR(nSample=200,n_components=10,refType=nclass,noise=1e-5)# estabilish PLS modeln_components=10plsdaModel=plsda(n_components=n_components)plsdaModel.fit(X,y)yhat=plsdaModel.predict(X)Feature selection# Feature selectionfrompynir.utilsimportsimulateNIRfrompynir.Calibrationimportplsfrompynir.FeatureSelectionimportMCUVE# simulate NIR dataX,y,wv=simulateNIR(nSample=200,n_components=10,noise=1e-5)mcModel=MCUVE(X,y,n_components,nrep=nrep).fit()featureSelected_MC_UVE=mcModel.featureRank[:nSel]Outlier dectionimportnumpyasnpimportmatplotlib.pyplotaspltfrompynir.utilsimportsimulateNIRfrompynir.OutlierDectionimportoutlierDection_PLS# simulate NIR dataX,y,wv=simulateNIR(nSample=200,n_components=10,noise=1e-5)ODModel=outlierDection_PLS(ncomp=3)Q,Tsq,Q_conf,Tsq_conf,idxOutlier=ODModel.fit(X,y).detect(X,y)ODModel.plot_HotellingT2_Q(Q,Tsq,Q_conf,Tsq_conf)Calibration Transferfrompynir.utilsimportsimulateNIR_calibrationTransferfrompynir.Calibrationimportpls,regresssionReportfrompynir.CalibrationTransferimportPDS,SST,BSimportmatplotlib.pyplotaspltimportnumpyasnp# Simulate NIR spectra for calibration transfernSample=100X1,X2,y,wv=simulateNIR_calibrationTransfer(nSample=nSample,n_components=10,shifts=5e1)idxTrain,idxTest=train_test_split(np.arange(nSample),test_size=0.6)idxTransfer,idxTest=train_test_split(idxTest,test_size=0.5)# Multivariate calibrationn_components=7plsModel1=pls(n_components=n_components).fit(X1[idxTrain,:],y[idxTrain])yhat1=plsModel1.predict(X1[idxTest,:],n_components=n_components)yhat2=plsModel1.predict(X2[idxTest,:],n_components=n_components)fig,ax=plt.subplots(2,sharex=True,figsize=(8,16))plsModel1.plot_prediction(y[idxTest],yhat1,title="First",ax=ax[0])plsModel1.plot_prediction(y[idxTest],yhat2,title="Second",ax=ax[1])# Calibration transfer on spectra## PDSX2_PDS=PDS(halfWindowSize=3).fit(X1[idxTransfer,:],X2[idxTransfer,:]).transform(X2[idxTest,:])yhat2_PDS=plsModel1.predict(X2_PDS,n_components=n_components)plsModel1.plot_prediction(y[idxTest],yhat2_PDS,title="PDS")fig,ax=plt.subplots()ax.plot(wv,np.transpose(X2_PDS))## SSTX2_SST=SST(n_components=n_components).fit(X1[idxTransfer,:],X2[idxTransfer,:]).transform(X2[idxTest,:])yhat2_SST=plsModel1.predict(X2_SST,n_components=n_components)plsModel1.plot_prediction(y[idxTest],yhat2_SST,title="SST")fig,ax=plt.subplots()ax.plot(wv,np.transpose(X2_SST))# Calibration transfer on prediction## BSyhat2_BS=BS().fit(yhat1,yhat2).transform(yhat2)plsModel1.plot_prediction(y[idxTest],yhat2_BS,title="BS")Calibration Enhancementfrompynir.utilsimportsimulateNIR_calibrationTransferfrompynir.Calibrationimportpls,regresssionReportfrompynir.CalibrationTransferimportNS_PFCE,SS_PFCE,FS_PFCE,MT_PFCEimportmatplotlib.pyplotaspltimportnumpyasnpfromsklearn.model_selectionimporttrain_test_splitimporttimenSample=100X1,X2,y,wv=simulateNIR_calibrationTransfer(nSample=nSample,n_components=10,shifts=5e1)idxTrain,idxTest=train_test_split(np.arange(nSample),test_size=0.6)idxTransfer,idxTest=train_test_split(idxTest,test_size=0.5)n_components=7plsModel1=pls(n_components=n_components).fit(X1[idxTrain,:],y[idxTrain])## PFCEthres=0.98constrType=1tic=time.time()b1=plsModel1.model['B'][:,-1]NS_PFCE_model=NS_PFCE(thres=thres,constrType=constrType).fit(X1[idxTransfer,:],X2[idxTransfer,:],b1)yhat2_NS_PFCE=NS_PFCE_model.transform(X2[idxTest,:])plsModel1.plot_prediction(y[idxTest],yhat2_NS_PFCE,title="NS-PFCE")fig,ax=plt.subplots()ax.plot(wv,NS_PFCE_model.b2.x[1:])ax.set_xlabel("wavelength (nm)")ax.set_ylabel("Regression Coefficients")ax.set_title("NS-PFCE")print("cost{:.2f}seconds for NS-PFCE".format(time.time()-tic))tic=time.time()b1=plsModel1.model['B'][:,-1]SS_PFCE_model=SS_PFCE(thres=thres,constrType=constrType).fit(X2[idxTransfer,:],y[idxTransfer],b1)yhat2_SS_PFCE=SS_PFCE_model.transform(X2[idxTest,:])plsModel1.plot_prediction(y[idxTest],yhat2_SS_PFCE,title="SS-PFCE")fig,ax=plt.subplots()ax.plot(wv,SS_PFCE_model.b2.x[1:])ax.set_xlabel("wavelength (nm)")ax.set_ylabel("Regression Coefficients")ax.set_title("SS-PFCE")print("cost{:.2f}seconds for SS-PFCE".format(time.time()-tic))tic=time.time()b1=plsModel1.model['B'][:,-1]FS_PFCE_model=FS_PFCE(thres=thres,constrType=constrType).fit(X1[idxTransfer,:],X2[idxTransfer,:],y[idxTransfer],b1)yhat2_FS_PFCE=FS_PFCE_model.transform(X2[idxTest,:])plsModel1.plot_prediction(y[idxTest],yhat2_FS_PFCE,title="FS-PFCE")fig,ax=plt.subplots()ax.plot(wv,FS_PFCE_model.b2.x[1:])ax.set_xlabel("wavelength (nm)")ax.set_ylabel("Regression Coefficients")ax.set_title("FS-PFCE")print("cost{:.2f}seconds for FS-PFCE".format(time.time()-tic))tic=time.time()b1=plsModel1.model['B'][:,-1]MT_PFCE_model=MT_PFCE(thres=thres,constrType=constrType)MT_PFCE_model.fit([X1[idxTrain,:],X2[idxTransfer,:]],(y[idxTrain],y[idxTransfer]),b1)yhat1_MT_PFCE=MT_PFCE_model.transform(X1[idxTest,:],0)# task 1yhat2_MT_PFCE=MT_PFCE_model.transform(X2[idxTest,:],1)# task 2fig,ax=plt.subplots(2,sharex=True,figsize=(8,16))plsModel1.plot_prediction(y[idxTest],yhat1_MT_PFCE,title="MT-PFCE_First",ax=ax[0])plsModel1.plot_prediction(y[idxTest],yhat2_MT_PFCE,title="MT-PFCE_Second",ax=ax[1])fig,ax=plt.subplots()ax.plot(wv,MT_PFCE_model.B.x.reshape(2,-1)[:,1:].transpose())ax.set_xlabel("wavelength (nm)")ax.set_ylabel("Regression Coefficients")ax.set_title("MT-PFCE")print("cost{:.2f}seconds for MT-PFCE".format(time.time()-tic))DemonFirst, executegitclonehttps://github.com/JinZhangLab/pynir.gitcd./pynir/examplesThen, execute code in your python coding environment or just in terminal as follows:pythonDemo1_SimulateNIR.py
pythonDemo2_Regression.py
pythonDemo3_Binary_Classification.py
pythonDemo4_Multiclass_Classification.py
pythonDemo5_dataPreprocessing.py
pythonDemo6_outierDection.py
pythonDemo7_FeatureSelection_oneStep
pythonDemo8_FeatureSelection_multiSteps.py
pythonDemo9_calibrationTransfer.py
pythonDemo10_calibrationTransfer_PFCE_simulateNIR.py
pythonDemo11_calibrationTransfer_PFCE_Tablet.py
pythonDemo12_calibrationTransfer_PFCE_Corn.pyRefZhang, J.; Cui, X. Y.; Cai, W. S.; Shao, X. G., A variable importance criterion for variable selection in near-infrared spectral analysis. Sci. China Chem. 2018, 62, 271-79.linkZhang J., Li B. Y., Hu Y., Zhou L. X., Wang G. Z., Guo G., Zhang Q. H., Lei S. C., Zhang A. H. A parameter-free framework for calibration enhancement of near-infrared spectroscopy based on correlation constraint [J]. Analytica Chimica Acta, 2021, 1142: 169-178.linkContributingPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.Please make sure to update tests as appropriate.LicenseMIT