Package: fast-tabnet
TabNet for fastaiThis is an adaptation of TabNet (Attention-based network for tabular data) for fastai (>=2.0) library. The original paperhttps://arxiv.org/pdf/1908.07442.pdf.Installpip install fast_tabnetHow to usemodel = TabNetModel(emb_szs, n_cont, out_sz, embed_p=0., y_range=None,  n_d=8, n_a=8, n_steps=3, gamma=1.5,  n_independent=2, n_shared=2, epsilon=1e-15, virtual_batch_size=128, momentum=0.02)Parametersemb_szs, n_cont, out_sz, embed_p, y_rangeare the same as for fastai TabularModel.n_d : int
Dimension of the prediction  layer (usually between 4 and 64)n_a : int
Dimension of the attention  layer (usually between 4 and 64)n_steps: int
Number of sucessive steps in the newtork (usually betwenn 3 and 10)gamma : float
Float above 1, scaling factor for attention updates (usually betwenn 1.0 to 2.0)momentum : float
Float value between 0 and 1 which will be used for momentum in all batch normn_independent : int
Number of independent GLU layer in each GLU block (default 2)n_shared : int
Number of independent GLU layer in each GLU block (default 2)epsilon: float
Avoid log(0), this should be kept very lowExampleBelow is an example from fastai library, but the model in use is TabNetfromfastai.basicsimport*fromfastai.tabular.allimport*fromfast_tabnet.coreimport*path=untar_data(URLs.ADULT_SAMPLE)df=pd.read_csv(path/'adult.csv')df_main,df_test=df.iloc[:-1000].copy(),df.iloc[-1000:].copy()df_main.head()<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>ageworkclassfnlwgteducationeducation-nummarital-statusoccupationrelationshipracesexcapital-gaincapital-losshours-per-weeknative-countrysalary049Private101320Assoc-acdm12.0Married-civ-spouseNaNWifeWhiteFemale0190240United-States>=50k144Private236746Masters14.0DivorcedExec-managerialNot-in-familyWhiteMale10520045United-States>=50k238Private96185HS-gradNaNDivorcedNaNUnmarriedBlackFemale0032United-States<50k338Self-emp-inc112847Prof-school15.0Married-civ-spouseProf-specialtyHusbandAsian-Pac-IslanderMale0040United-States>=50k442Self-emp-not-inc822977th-8thNaNMarried-civ-spouseOther-serviceWifeBlackFemale0050United-States<50kcat_names=['workclass','education','marital-status','occupation','relationship','race','native-country','sex']cont_names=['age','fnlwgt','education-num']procs=[Categorify,FillMissing,Normalize]splits=RandomSplitter()(range_of(df_main))to=TabularPandas(df_main,procs,cat_names,cont_names,y_names="salary",y_block=CategoryBlock(),splits=splits)dls=to.dataloaders(bs=32)dls.valid.show_batch()workclasseducationmarital-statusoccupationrelationshipracenative-countrysexeducation-num_naagefnlwgteducation-numsalary0PrivateHS-gradMarried-civ-spouseOther-serviceWifeWhiteUnited-StatesFemaleFalse39.000000196673.0001159.0<50k1PrivateHS-gradMarried-civ-spouseCraft-repairHusbandWhiteUnited-StatesMaleFalse32.000000198067.9997719.0<50k2State-govHS-gradNever-marriedAdm-clericalOwn-childWhiteUnited-StatesFemaleFalse18.999999176633.9999319.0<50k3PrivateSome-collegeMarried-civ-spouseProf-specialtyHusbandWhiteUnited-StatesMaleFalse67.999999107626.99849010.0<50k4PrivateMastersNever-marriedExec-managerialNot-in-familyBlackUnited-StatesMaleFalse29.000000214925.00026014.0<50k5PrivateHS-gradMarried-civ-spousePriv-house-servWifeWhiteUnited-StatesFemaleFalse22.000000200109.0001269.0<50k6PrivateSome-collegeNever-marriedSalesOwn-childWhiteUnited-StatesFemaleFalse18.00000060980.99842910.0<50k7PrivateSome-collegeSeparatedAdm-clericalNot-in-familyWhiteUnited-StatesFemaleFalse28.000000334367.99819910.0<50k8Private11thMarried-civ-spouseTransport-movingHusbandWhiteUnited-StatesMaleFalse49.000000123584.0010977.0<50k9PrivateMastersNever-marriedProf-specialtyNot-in-familyWhiteUnited-StatesFemaleFalse26.000000397316.99992214.0<50kto_tst=to.new(df_test)to_tst.process()to_tst.all_cols.head()<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>workclasseducationmarital-statusoccupationrelationshipracenative-countrysexeducation-num_naagefnlwgteducation-numsalary315615259334021-1.505833-0.559418-1.20217003156221252534011-1.4326530.421241-0.4180320315635734154021-0.1154060.132868-1.986307031564812391540211.4945610.749805-0.41803203156511211534021-0.4813087.529798-0.4180320emb_szs=get_emb_sz(to)That's the use of the modelmodel=TabNetModel(emb_szs,len(to.cont_names),dls.c,n_d=8,n_a=8,n_steps=5,mask_type='entmax');learn=Learner(dls,model,CrossEntropyLossFlat(),opt_func=Adam,lr=3e-2,metrics=[accuracy])learn.lr_find()SuggestedLRs(lr_min=0.2754228591918945, lr_steep=1.9054607491852948e-06)learn.fit_one_cycle(5)epochtrain_lossvalid_lossaccuracytime00.4462740.4144510.81701500:3010.3800020.3930300.81891600:3020.3711490.3598020.83206600:3030.3490270.3522550.83586800:3040.3553390.3493600.83681900:30Tabnet interpretability# feature importance for 2k rowsdl=learn.dls.test_dl(df.iloc[:2000],bs=256)feature_importances=tabnet_feature_importances(learn.model,dl)# per sample interpretationdl=learn.dls.test_dl(df.iloc[:20],bs=20)res_explain,res_masks=tabnet_explain(learn.model,dl)plt.xticks(rotation='vertical')plt.bar(dl.x_names,feature_importances,color='g')plt.show()defplot_explain(masks,lbls,figsize=(12,12)):"Plots masks with `lbls` (`dls.x_names`)"fig=plt.figure(figsize=figsize)ax=fig.add_axes([0.1,0.1,0.8,0.8])plt.yticks(np.arange(0,len(masks),1.0))plt.xticks(np.arange(0,len(masks[0]),1.0))ax.set_xticklabels(lbls,rotation=90)plt.ylabel('Sample Number')plt.xlabel('Variable')plt.imshow(masks)plot_explain(res_explain,dl.x_names)Hyperparameter search with Bayesian OptimizationIf your dataset isn't huge you can tune hyperparameters for tabular models with Bayesian Optimization. You can optimize directly your metric using this approach if the metric is sensitive enough (in our example it is not and we use validation loss instead). Also, you should create the second validation set, because you will use the first as a training set for Bayesian Optimization.You may need to install the optimizerpip install bayesian-optimizationfromfunctoolsimportlru_cache# The function we'll optimize@lru_cache(1000)defget_accuracy(n_d:Int,n_a:Int,n_steps:Int):model=TabNetModel(emb_szs,len(to.cont_names),dls.c,n_d=n_d,n_a=n_a,n_steps=n_steps,gamma=1.5)learn=Learner(dls,model,CrossEntropyLossFlat(),opt_func=opt_func,lr=3e-2,metrics=[accuracy])learn.fit_one_cycle(5)returnfloat(learn.validate(dl=learn.dls.valid)[1])This implementation of Bayesian Optimization doesn't work naturally with descreet values. That's why we use wrapper withlru_cache.deffit_accuracy(pow_n_d,pow_n_a,pow_n_steps):n_d,n_a,n_steps=map(lambdax:2**int(x),(pow_n_d,pow_n_a,pow_n_steps))returnget_accuracy(n_d,n_a,n_steps)frombayes_optimportBayesianOptimization# Bounded region of parameter spacepbounds={'pow_n_d':(0,8),'pow_n_a':(0,8),'pow_n_steps':(0,4)}optimizer=BayesianOptimization(f=fit_accuracy,pbounds=pbounds,)optimizer.maximize(init_points=15,n_iter=100,)|   iter    |  target   |  pow_n_a  |  pow_n_d  | pow_n_... |
-------------------------------------------------------------epochtrain_lossvalid_lossaccuracytime00.4048880.4328340.79388500:1010.3679790.3848400.81860000:0920.3664440.3720050.81970800:0930.3627710.3669490.82351100:1040.3536820.3671320.82351100:10| [0m 1       [0m | [0m 0.8235  [0m | [0m 0.9408  [0m | [0m 1.898   [0m | [0m 1.652   [0m |epochtrain_lossvalid_lossaccuracytime00.3933010.4497420.81083600:0810.3791400.4137730.81558900:0720.3557900.3889070.82256000:0730.3499840.3626710.82873900:0740.3480000.3601500.82731300:07| [95m 2       [0m | [95m 0.8273  [0m | [95m 4.262   [0m | [95m 5.604   [0m | [95m 0.2437  [0m |epochtrain_lossvalid_lossaccuracytime00.4515720.4341890.78121000:1210.4237630.4134200.80545000:1220.3989220.4086880.81416400:1230.3909810.3923980.80893500:1240.3764180.3822500.81717400:12| [0m 3       [0m | [0m 0.8172  [0m | [0m 7.233   [0m | [0m 6.471   [0m | [0m 2.508   [0m |epochtrain_lossvalid_lossaccuracytime00.4031870.4139860.79816200:0710.3985440.3901020.82018400:0720.3905690.3897030.82525300:0730.3754260.3857060.82699600:0740.3704460.3833660.83111500:06| [95m 4       [0m | [95m 0.8311  [0m | [95m 5.935   [0m | [95m 1.241   [0m | [95m 0.3809  [0m |epochtrain_lossvalid_lossaccuracytime00.4641450.4586410.75126700:1810.4246910.4369680.78802300:1820.4315760.4365810.77582400:1830.4321430.4370620.75950600:1840.4299150.4383320.75855500:18| [0m 5       [0m | [0m 0.7586  [0m | [0m 2.554   [0m | [0m 0.4992  [0m | [0m 3.111   [0m |epochtrain_lossvalid_lossaccuracytime00.4703590.4758260.74889100:1210.4115640.4094330.79705300:1220.3927180.3973630.80972700:1230.3875640.3800330.81432200:1240.3741530.3782580.81891600:12| [0m 6       [0m | [0m 0.8189  [0m | [0m 4.592   [0m | [0m 2.138   [0m | [0m 2.824   [0m |epochtrain_lossvalid_lossaccuracytime00.5470420.5887520.75411900:1810.4917310.4697950.77186300:1820.4543400.4339610.77519000:1830.4243860.4323850.78295300:1840.3976450.4064200.80576700:19| [0m 7       [0m | [0m 0.8058  [0m | [0m 6.186   [0m | [0m 7.016   [0m | [0m 3.316   [0m |epochtrain_lossvalid_lossaccuracytime00.4852450.4876350.75110900:1810.4508320.4464230.75031700:1820.4482030.4494190.75522800:1830.4302580.4435620.74429700:1840.4298210.4371730.76156500:18| [0m 8       [0m | [0m 0.7616  [0m | [0m 2.018   [0m | [0m 1.316   [0m | [0m 3.675   [0m |epochtrain_lossvalid_lossaccuracytime00.4584250.4557330.75158400:1210.4397810.4678070.75110900:1220.4203310.4322160.77519000:1230.4210120.4214120.78231900:1240.4018280.4134340.80101400:12| [0m 9       [0m | [0m 0.801   [0m | [0m 2.051   [0m | [0m 1.958   [0m | [0m 2.332   [0m |epochtrain_lossvalid_lossaccuracytime00.5469970.5067280.76140700:1810.4897120.4393240.79958800:1820.4485580.4484190.78612200:1830.4368690.4353750.80164800:1840.4171280.4210930.79832100:18| [0m 10      [0m | [0m 0.7983  [0m | [0m 5.203   [0m | [0m 7.719   [0m | [0m 3.407   [0m |epochtrain_lossvalid_lossaccuracytime00.3807810.4634090.78643900:0710.3592120.4611470.79832100:0720.3514140.3689500.82271900:0730.3472570.3670560.82937300:0740.3372120.3623750.83079900:07| [0m 11      [0m | [0m 0.8308  [0m | [0m 6.048   [0m | [0m 4.376   [0m | [0m 0.08141 [0m |epochtrain_lossvalid_lossaccuracytime00.4307720.4308970.76774400:1210.4026110.4321370.76425900:1220.4075790.4096510.81210400:1230.3749880.3918220.81669800:1240.3780110.3892780.81606500:12| [0m 12      [0m | [0m 0.8161  [0m | [0m 7.083   [0m | [0m 1.385   [0m | [0m 2.806   [0m |epochtrain_lossvalid_lossaccuracytime00.4020180.4120510.81226200:0910.3728040.4649370.81162900:0920.3682740.3846750.82018400:0930.3645020.3719200.82065900:0940.3489980.3694450.82382800:09| [0m 13      [0m | [0m 0.8238  [0m | [0m 4.812   [0m | [0m 3.785   [0m | [0m 1.396   [0m |
| [0m 14      [0m | [0m 0.8172  [0m | [0m 7.672   [0m | [0m 6.719   [0m | [0m 2.72    [0m |epochtrain_lossvalid_lossaccuracytime00.4760330.4425980.80354900:1210.4052360.4140150.78897300:1120.4062910.4512690.78944900:1130.3910130.3932430.81606500:1240.3741600.3776350.82145100:12| [0m 15      [0m | [0m 0.8215  [0m | [0m 6.464   [0m | [0m 7.954   [0m | [0m 2.647   [0m |epochtrain_lossvalid_lossaccuracytime00.3901420.3906780.81099500:0610.3817170.3822020.81305500:0620.3685640.3787050.82382800:0630.3588580.3683290.82351100:0740.3533920.3639130.82588700:06| [0m 16      [0m | [0m 0.8259  [0m | [0m 0.1229  [0m | [0m 7.83    [0m | [0m 0.3708  [0m |epochtrain_lossvalid_lossaccuracytime00.3812150.4226510.80069700:0610.3773450.3808630.81590600:0620.3666310.3705790.82287700:0630.3627450.3666190.82335200:0740.3568610.3648350.82588700:07| [0m 17      [0m | [0m 0.8259  [0m | [0m 0.03098 [0m | [0m 3.326   [0m | [0m 0.007025[0m |epochtrain_lossvalid_lossaccuracytime00.4046040.4430350.82446100:0710.3618720.3888800.82366900:0620.3751640.3699680.82509500:0630.3520910.3638230.82794700:0640.3354580.3625440.82937300:07| [0m 18      [0m | [0m 0.8294  [0m | [0m 7.81    [0m | [0m 7.976   [0m | [0m 0.0194  [0m |epochtrain_lossvalid_lossaccuracytime00.6792920.6772990.24889100:0510.6754030.6784060.24889100:0520.6732590.6733740.24889100:0630.6749960.6735140.24889100:0740.6688130.6736710.24889100:07| [0m 19      [0m | [0m 0.2489  [0m | [0m 0.4499  [0m | [0m 0.138   [0m | [0m 0.001101[0m |epochtrain_lossvalid_lossaccuracytime00.5242010.5281320.72988000:3010.4193770.4031980.81210400:3120.3993980.4188900.81242100:3130.3816510.3917440.81907500:3140.3687420.3779040.82208500:31| [0m 20      [0m | [0m 0.8221  [0m | [0m 0.0     [0m | [0m 6.575   [0m | [0m 4.0     [0m |epochtrain_lossvalid_lossaccuracytime00.6810830.6823970.24889100:0510.6729350.6793710.24889100:0620.6752000.6734660.24889100:0630.6742510.6733560.24889100:0640.6688610.6731860.24889100:06| [0m 21      [0m | [0m 0.2489  [0m | [0m 8.0     [0m | [0m 0.0     [0m | [0m 0.0     [0m |epochtrain_lossvalid_lossaccuracytime00.4072460.4322030.80133100:1010.3850860.3995130.81131200:1120.3773650.3841210.81606500:1230.3668550.3710100.82319400:1240.3619310.3689330.82509500:12| [0m 22      [0m | [0m 0.8251  [0m | [0m 0.0     [0m | [0m 4.502   [0m | [0m 2.193   [0m |epochtrain_lossvalid_lossaccuracytime00.4936230.4769210.76600100:3010.4411260.4437740.77677400:3120.4245230.4371250.78390400:3130.4024570.4086280.79594400:3140.4394200.4317560.78897300:32| [0m 23      [0m | [0m 0.789   [0m | [0m 8.0     [0m | [0m 3.702   [0m | [0m 4.0     [0m |epochtrain_lossvalid_lossaccuracytime00.5156150.5139190.75110900:3110.4626740.4953220.75158400:3120.4654300.4836850.75126700:3130.4813080.4953750.75507000:3140.4813240.4912750.75491100:32| [0m 24      [0m | [0m 0.7549  [0m | [0m 6.009   [0m | [0m 0.0     [0m | [0m 4.0     [0m |epochtrain_lossvalid_lossaccuracytime00.4228370.4039530.81939200:0610.3807530.3673450.82683800:0620.3530450.3651740.83000600:0730.3486280.3642820.82636200:0740.3435610.3615090.82921400:07| [0m 25      [0m | [0m 0.8292  [0m | [0m 3.522   [0m | [0m 8.0     [0m | [0m 0.0     [0m |epochtrain_lossvalid_lossaccuracytime00.8077661.3072790.48162200:3110.5133080.4994700.78358700:3220.4459060.4926200.79800400:3130.3850940.3999860.80750900:3240.3872280.3847390.81701500:31| [0m 26      [0m | [0m 0.817   [0m | [0m 0.0     [0m | [0m 8.0     [0m | [0m 4.0     [0m |epochtrain_lossvalid_lossaccuracytime00.4420760.4913380.75538700:3110.4410780.4436740.76077300:3120.4175750.4187580.79214200:3130.4108250.4175810.78849800:3440.4034070.4109410.79832100:46| [0m 27      [0m | [0m 0.7983  [0m | [0m 0.0     [0m | [0m 0.0     [0m | [0m 4.0     [0m |epochtrain_lossvalid_lossaccuracytime00.4070060.4196790.79214200:0810.3909130.3926310.81052000:0820.3655600.3943300.81749100:0830.3784590.3872440.82065900:0840.3752750.3854170.82889700:08| [0m 28      [0m | [0m 0.8289  [0m | [0m 3.379   [0m | [0m 2.848   [0m | [0m 0.0     [0m |epochtrain_lossvalid_lossaccuracytime00.4306040.4695920.78121000:4510.4230740.4297040.79752900:4520.4001200.3933980.81099500:4530.3823610.3906510.81606500:4640.3895200.4018780.80719300:46| [0m 29      [0m | [0m 0.8072  [0m | [0m 0.0     [0m | [0m 2.588   [0m | [0m 4.0     [0m |epochtrain_lossvalid_lossaccuracytime00.3963480.3974540.80671700:0810.3833420.3860230.81955000:0720.3694930.3744010.82002500:0730.3560150.3665350.82620400:0840.3410730.3652410.82620400:08| [0m 30      [0m | [0m 0.8262  [0m | [0m 1.217   [0m | [0m 5.622   [0m | [0m 0.0     [0m |epochtrain_lossvalid_lossaccuracytime00.8090770.4805910.78279500:4510.5713180.4977310.73906800:4520.5145620.4617260.78152700:4530.4398220.4517220.78723100:4440.4198810.4221250.80164800:45| [0m 31      [0m | [0m 0.8016  [0m | [0m 8.0     [0m | [0m 8.0     [0m | [0m 4.0     [0m |epochtrain_lossvalid_lossaccuracytime00.4105210.4350450.81004400:0810.3630980.3780010.82192600:0820.3595250.3644770.82778800:0830.3540050.3665070.82161000:0840.3472930.3626570.82937300:08| [0m 32      [0m | [0m 0.8294  [0m | [0m 5.864   [0m | [0m 8.0     [0m | [0m 0.0     [0m |epochtrain_lossvalid_lossaccuracytime00.4983760.4366960.79404300:1610.4116990.4355370.80133100:1620.3853270.3969160.82018400:1630.3820200.3898560.81337100:1640.3738690.3778040.82081700:15| [0m 33      [0m | [0m 0.8208  [0m | [0m 1.776   [0m | [0m 8.0     [0m | [0m 2.212   [0m |epochtrain_lossvalid_lossaccuracytime00.4046530.4401060.77218000:1110.3779310.3937150.81733200:1120.3732210.3792730.82683800:1130.3596820.3628440.82842200:1140.3403840.3630720.82889700:11| [0m 34      [0m | [0m 0.8289  [0m | [0m 5.777   [0m | [0m 2.2     [0m | [0m 1.31    [0m |epochtrain_lossvalid_lossaccuracytime00.5203080.5032070.74920800:4510.4725010.4514690.78041800:4520.4546860.4291750.78485400:4530.4008000.4137270.79546900:4440.4056040.4097700.80164800:45| [0m 35      [0m | [0m 0.8016  [0m | [0m 2.748   [0m | [0m 5.915   [0m | [0m 4.0     [0m |epochtrain_lossvalid_lossaccuracytime00.5045370.5015410.75031700:4510.4659370.4777150.77328900:4520.4353640.4814150.76663500:4530.4254340.4421980.77281400:4540.4257790.4589470.77186300:45| [0m 36      [0m | [0m 0.7719  [0m | [0m 6.251   [0m | [0m 2.532   [0m | [0m 4.0     [0m |epochtrain_lossvalid_lossaccuracytime00.4207820.4207210.79135000:1010.4035760.4083760.80022200:1020.3902360.3936240.82034200:1130.3777770.3896570.82161000:1140.3828090.3860110.82097600:11| [0m 37      [0m | [0m 0.821   [0m | [0m 5.093   [0m | [0m 0.172   [0m | [0m 1.64    [0m |epochtrain_lossvalid_lossaccuracytime00.3935750.3978110.81226200:0810.3782720.3819150.81574800:0820.3647990.3692140.82462000:0830.3557570.3645540.82699600:0840.3420900.3627230.82430300:08| [0m 38      [0m | [0m 0.8243  [0m | [0m 8.0     [0m | [0m 5.799   [0m | [0m 0.0     [0m |epochtrain_lossvalid_lossaccuracytime00.3936930.3969800.82208500:1110.3612310.3931460.81384700:1120.3456450.3795100.82398600:1130.3497780.3670770.82667900:1140.3423900.3620270.82778800:11| [0m 39      [0m | [0m 0.8278  [0m | [0m 1.62    [0m | [0m 3.832   [0m | [0m 1.151   [0m |epochtrain_lossvalid_lossaccuracytime00.8327370.4910020.77154600:4310.6279480.5535520.76473400:4320.4989010.4671620.79198400:4630.4311960.4445760.78564600:4340.3997450.4270600.79657800:42| [0m 40      [0m | [0m 0.7966  [0m | [0m 2.198   [0m | [0m 8.0     [0m | [0m 4.0     [0m |epochtrain_lossvalid_lossaccuracytime00.5113010.5144010.75126700:4310.4473320.4451570.75110900:4320.4511250.4383270.75095100:4230.4458830.4432660.75126700:4240.4448160.4384590.76410000:42| [0m 41      [0m | [0m 0.7641  [0m | [0m 8.0     [0m | [0m 1.03    [0m | [0m 4.0     [0m |epochtrain_lossvalid_lossaccuracytime00.4085040.4132750.79721200:1510.3927070.3990850.80576700:1520.3799380.3955500.81780700:1530.3752880.3831860.82081700:1540.3604170.3750980.82319400:16| [0m 42      [0m | [0m 0.8232  [0m | [0m 0.0     [0m | [0m 2.504   [0m | [0m 2.135   [0m |epochtrain_lossvalid_lossaccuracytime00.3993710.4151960.80101400:0710.3678040.3920200.81099500:0620.3622880.3851240.82065900:0730.3447280.3713390.82366900:0740.3457690.3620590.82937300:07| [0m 43      [0m | [0m 0.8294  [0m | [0m 0.0     [0m | [0m 5.441   [0m | [0m 0.0     [0m |epochtrain_lossvalid_lossaccuracytime00.3971570.4310030.80386600:0610.3949640.3964480.81036100:0620.3785840.3879430.82065900:0730.3716010.3861860.81828300:0740.3697590.3843390.82763000:07| [0m 44      [0m | [0m 0.8276  [0m | [0m 4.636   [0m | [0m 1.476   [0m | [0m 0.0     [0m |epochtrain_lossvalid_lossaccuracytime00.4086540.4268060.79119100:1210.3941840.4065860.78643900:1220.3696250.3726800.82256000:1230.3494440.3681420.82382800:1240.3516840.3634060.82620400:12| [0m 45      [0m | [0m 0.8262  [0m | [0m 0.0     [0m | [0m 7.071   [0m | [0m 2.071   [0m |epochtrain_lossvalid_lossaccuracytime00.4002930.4160980.81162900:0810.3773870.4333950.80703400:0820.3681310.3954480.79642000:0830.3677500.3768790.81717400:0840.3621240.3714320.82113400:08| [0m 46      [0m | [0m 0.8211  [0m | [0m 4.26    [0m | [0m 6.934   [0m | [0m 1.79    [0m |epochtrain_lossvalid_lossaccuracytime00.4045790.4374430.81479700:0710.3753420.3804160.82493700:0720.3658350.3776170.81273800:0730.3546190.3645030.82747100:0740.3406030.3634880.82794700:07| [0m 47      [0m | [0m 0.8279  [0m | [0m 6.579   [0m | [0m 6.485   [0m | [0m 0.0     [0m |epochtrain_lossvalid_lossaccuracytime00.3848900.4403420.81257900:0810.3714830.3872000.81384700:0920.3659510.3780710.81828300:0930.3629650.3699940.82161000:0940.3564830.3651510.82652100:09| [0m 48      [0m | [0m 0.8265  [0m | [0m 8.0     [0m | [0m 4.293   [0m | [0m 1.74    [0m |epochtrain_lossvalid_lossaccuracytime00.3863080.3892500.81543100:0810.3684020.3893380.81495600:0920.3622110.3771960.82477800:0930.3561350.3629510.82953100:0940.3415770.3624760.83079900:09| [0m 49      [0m | [0m 0.8308  [0m | [0m 7.909   [0m | [0m 7.827   [0m | [0m 1.323   [0m |epochtrain_lossvalid_lossaccuracytime00.4260440.4228820.79198400:0810.3757560.3818100.81749100:0920.3639320.3759040.81891600:0930.3494420.3650520.82398600:0940.3445090.3630270.83032300:09| [0m 50      [0m | [0m 0.8303  [0m | [0m 4.946   [0m | [0m 1.246   [0m | [0m 1.589   [0m |epochtrain_lossvalid_lossaccuracytime00.3885220.4319090.82097600:0710.3725320.4486440.75110900:0720.3588230.3733220.82366900:0730.3528380.3624240.83159100:0740.3529490.3613560.83143200:07| [95m 51      [0m | [95m 0.8314  [0m | [95m 5.664   [0m | [95m 2.626   [0m | [95m 0.003048[0m |epochtrain_lossvalid_lossaccuracytime00.3891950.3900320.81733200:0610.3699930.3821990.81970800:0720.3628010.3732820.82652100:0630.3597600.3635970.82430300:0640.3445250.3620970.82889700:07| [0m 52      [0m | [0m 0.8289  [0m | [0m 1.287   [0m | [0m 3.505   [0m | [0m 0.06804 [0m |epochtrain_lossvalid_lossaccuracytime00.3949400.4031650.81463900:0610.3715180.4521180.80687600:0620.3647340.3772140.82446100:0630.3479680.3653350.82351100:0740.3454760.3636700.82715500:07| [0m 53      [0m | [0m 0.8272  [0m | [0m 1.606   [0m | [0m 7.998   [0m | [0m 0.2009  [0m |epochtrain_lossvalid_lossaccuracytime00.5003220.5192610.75712900:1010.4132700.4236300.80196500:1120.3802340.3955880.81337100:1230.3616770.3781230.81717400:1240.3746290.3737720.82002500:12| [0m 54      [0m | [0m 0.82    [0m | [0m 4.579   [0m | [0m 5.017   [0m | [0m 2.928   [0m |
| [0m 55      [0m | [0m 0.8259  [0m | [0m 0.02565 [0m | [0m 3.699   [0m | [0m 0.9808  [0m |epochtrain_lossvalid_lossaccuracytime00.4527870.4436970.76869500:1110.4283320.4154540.80069700:1120.3965220.4028500.80766800:1230.4248020.4146480.78358700:1240.3850550.3923590.80148900:12| [0m 56      [0m | [0m 0.8015  [0m | [0m 1.927   [0m | [0m 5.92    [0m | [0m 2.53    [0m |epochtrain_lossvalid_lossaccuracytime00.4355970.4382220.81083600:1910.3999200.5311890.77075400:1920.4034080.4093820.80481600:1830.3635190.3838230.81590600:1940.3600300.3776210.81970800:19| [0m 57      [0m | [0m 0.8197  [0m | [0m 0.7796  [0m | [0m 4.576   [0m | [0m 3.952   [0m |epochtrain_lossvalid_lossaccuracytime00.3884450.4202430.80053900:0710.3729120.3696590.82763000:0720.3544430.3667570.82810500:0730.3524680.3660380.82256000:0740.3478220.3620010.82969000:07| [0m 58      [0m | [0m 0.8297  [0m | [0m 3.525   [0m | [0m 4.198   [0m | [0m 0.02314 [0m |epochtrain_lossvalid_lossaccuracytime00.4446270.4317390.78707200:1210.3926370.4129850.79974700:1220.3697330.3961330.80244000:1230.3658210.3730950.82034200:1240.3704860.3715600.81939200:12| [0m 59      [0m | [0m 0.8194  [0m | [0m 6.711   [0m | [0m 3.848   [0m | [0m 2.395   [0m |epochtrain_lossvalid_lossaccuracytime00.3968310.3890450.80956900:0710.3711710.3750650.81860000:0720.3503090.3717950.82462000:0730.3597000.3630410.82873900:0740.3457350.3615560.83079900:07| [0m 60      [0m | [0m 0.8308  [0m | [0m 4.914   [0m | [0m 7.944   [0m | [0m 0.9998  [0m |epochtrain_lossvalid_lossaccuracytime00.4228530.4126910.80434100:0910.3752090.3946920.81717400:0920.3655740.3803760.82018400:0830.3591430.3636070.83111500:0840.3479910.3616500.82794700:08| [0m 61      [0m | [0m 0.8279  [0m | [0m 7.962   [0m | [0m 6.151   [0m | [0m 1.119   [0m |epochtrain_lossvalid_lossaccuracytime00.3881470.4058850.81067800:0710.3677430.3918670.80782600:0720.3669640.3629800.82873900:0730.3634020.3633960.82953100:0740.3510940.3622450.82921400:07| [0m 62      [0m | [0m 0.8292  [0m | [0m 2.583   [0m | [0m 6.996   [0m | [0m 0.008348[0m |epochtrain_lossvalid_lossaccuracytime00.4327240.3905160.81511400:1110.4071000.4015640.81115300:1120.3594140.3844630.82034200:1130.3580610.3718440.82636200:1240.3453570.3629860.83111500:12| [0m 63      [0m | [0m 0.8311  [0m | [0m 0.2249  [0m | [0m 8.0     [0m | [0m 2.823   [0m |epochtrain_lossvalid_lossaccuracytime00.4189900.4204630.80861800:0710.3898300.3981100.81622300:0820.3829750.3876200.81495600:0630.3840930.3796070.81939200:0640.3580190.3711400.82382800:08| [0m 64      [0m | [0m 0.8238  [0m | [0m 5.764   [0m | [0m 5.509   [0m | [0m 1.482   [0m |epochtrain_lossvalid_lossaccuracytime00.3851470.3996800.80624200:0910.3760320.3811310.82256000:0920.3638700.3782270.82240200:0930.3510890.3687900.82683800:0940.3404040.3618070.82921400:09| [0m 65      [0m | [0m 0.8292  [0m | [0m 1.048   [0m | [0m 2.939   [0m | [0m 1.922   [0m |epochtrain_lossvalid_lossaccuracytime00.5266280.5072360.74667300:1710.4602290.4556750.76568400:1920.4174270.4213680.78596300:1930.4628000.4588440.77392300:1940.4494790.4566270.78358700:19| [0m 66      [0m | [0m 0.7836  [0m | [0m 3.68    [0m | [0m 3.977   [0m | [0m 3.919   [0m |epochtrain_lossvalid_lossaccuracytime00.6216780.5590800.74904900:1010.4571040.4736100.75839700:1120.4162870.4166220.76457500:1330.3881070.4038440.81194500:1340.3842310.3963970.81305500:13| [0m 67      [0m | [0m 0.8131  [0m | [0m 5.907   [0m | [0m 0.9452  [0m | [0m 2.168   [0m |epochtrain_lossvalid_lossaccuracytime00.4101050.4161710.80861800:1110.3816690.4001090.80909400:1320.3775390.4038790.80307400:1330.3746530.3891220.80861800:1340.3663560.3805260.81400500:13| [0m 68      [0m | [0m 0.814   [0m | [0m 7.981   [0m | [0m 2.796   [0m | [0m 2.78    [0m |epochtrain_lossvalid_lossaccuracytime00.3805390.4208560.81558900:0710.3682320.4242760.80386600:0520.3581940.3785010.82715500:0630.3534270.3622240.82969000:0740.3444430.3615540.82683800:07| [0m 69      [0m | [0m 0.8268  [0m | [0m 7.223   [0m | [0m 3.762   [0m | [0m 0.6961  [0m |epochtrain_lossvalid_lossaccuracytime00.4459240.4885810.75269300:1710.4107090.4009620.81368800:1820.3735180.3932350.82018400:1830.3641600.3789200.82081700:1740.3575510.3716290.82541200:17| [0m 70      [0m | [0m 0.8254  [0m | [0m 0.009375[0m | [0m 5.081   [0m | [0m 3.79    [0m |epochtrain_lossvalid_lossaccuracytime00.4103230.4426700.81479700:0810.3792790.4055000.80703400:0820.3875760.3924480.81970800:0930.3716220.3891670.82303500:0940.3741820.3869640.82509500:09| [0m 71      [0m | [0m 0.8251  [0m | [0m 3.293   [0m | [0m 2.76    [0m | [0m 1.061   [0m |
| [0m 72      [0m | [0m 0.8308  [0m | [0m 4.589   [0m | [0m 7.13    [0m | [0m 0.003179[0m |epochtrain_lossvalid_lossaccuracytime00.3915090.3999660.80640000:0610.3666940.4057190.82382800:0720.3597510.3754960.82287700:0730.3476780.3617110.83079900:0740.3368960.3619220.82858000:07| [0m 73      [0m | [0m 0.8286  [0m | [0m 7.118   [0m | [0m 5.204   [0m | [0m 0.5939  [0m |epochtrain_lossvalid_lossaccuracytime00.4354850.4148830.80814300:0810.3735910.4171380.81400500:0920.3695900.3757240.82018400:0930.3708290.3686550.82953100:0940.3464630.3663070.82541200:09| [0m 74      [0m | [0m 0.8254  [0m | [0m 6.837   [0m | [0m 7.988   [0m | [0m 1.055   [0m |
| [0m 75      [0m | [0m 0.8259  [0m | [0m 0.6629  [0m | [0m 7.012   [0m | [0m 0.03222 [0m |
| [0m 76      [0m | [0m 0.8311  [0m | [0m 5.177   [0m | [0m 1.457   [0m | [0m 0.5857  [0m |epochtrain_lossvalid_lossaccuracytime00.4557410.5300080.79499400:1110.4219610.4233170.80529200:1120.4057990.4057290.80735100:1230.3838950.3950920.81685700:1240.3788820.3860440.81875800:12| [0m 77      [0m | [0m 0.8188  [0m | [0m 3.308   [0m | [0m 4.533   [0m | [0m 2.048   [0m |epochtrain_lossvalid_lossaccuracytime00.4122610.4109500.79895400:0810.3823140.4084710.80339000:0920.3474880.3875000.81527300:0930.3434080.3720500.82145100:0940.3449630.3661580.82271900:09| [0m 78      [0m | [0m 0.8227  [0m | [0m 0.4036  [0m | [0m 7.997   [0m | [0m 1.439   [0m |epochtrain_lossvalid_lossaccuracytime00.4128560.4100160.79911300:0810.4108520.4164050.78849800:0820.3738970.3843850.82430300:0930.3531640.3661290.82271900:0940.3532530.3622690.82636200:09| [0m 79      [0m | [0m 0.8264  [0m | [0m 3.438   [0m | [0m 7.982   [0m | [0m 1.829   [0m |epochtrain_lossvalid_lossaccuracytime00.4193160.4089360.79816200:0810.3938260.3905260.82018400:0820.3728790.3748230.82271900:0830.3580190.3709130.82034200:0840.3460200.3622520.82969000:08| [0m 80      [0m | [0m 0.8297  [0m | [0m 6.88    [0m | [0m 2.404   [0m | [0m 1.666   [0m |epochtrain_lossvalid_lossaccuracytime00.4334810.4373200.79008200:1810.4152800.4029460.81416400:1820.3655750.3762850.82287700:1830.3632060.3718650.82050100:1840.3564010.3702520.82382800:18| [0m 81      [0m | [0m 0.8238  [0m | [0m 0.03221 [0m | [0m 1.306   [0m | [0m 3.909   [0m |epochtrain_lossvalid_lossaccuracytime00.3931500.4209640.78374500:0710.3750650.3713800.82398600:0720.3629520.3870370.81368800:0730.3472450.3702250.82493700:0740.3484060.3614200.83064000:07| [0m 82      [0m | [0m 0.8306  [0m | [0m 1.575   [0m | [0m 2.689   [0m | [0m 0.8684  [0m |epochtrain_lossvalid_lossaccuracytime00.3955300.3974300.81860000:0610.3586790.3967730.81828300:0720.3493050.3728770.82382800:0730.3473460.3630060.82842200:0740.3356520.3625670.83095700:07| [0m 83      [0m | [0m 0.831   [0m | [0m 2.765   [0m | [0m 5.439   [0m | [0m 0.04047 [0m |epochtrain_lossvalid_lossaccuracytime00.3808310.3860720.81432200:0510.3697780.3925210.81052000:0720.3682860.3831310.81685700:0730.3565850.3678390.82176800:0740.3447220.3666390.82525300:07| [0m 84      [0m | [0m 0.8253  [0m | [0m 0.1961  [0m | [0m 4.123   [0m | [0m 0.02039 [0m |epochtrain_lossvalid_lossaccuracytime00.5026050.4637030.77218000:1110.4072310.4043860.80449900:1320.4068490.4112540.81749100:1230.3799100.3891180.81717400:1240.3709640.3798350.82129300:12| [0m 85      [0m | [0m 0.8213  [0m | [0m 7.937   [0m | [0m 7.939   [0m | [0m 2.895   [0m |epochtrain_lossvalid_lossaccuracytime00.4025740.4312370.78675500:1110.3804940.3922530.81796600:1120.3632840.3939070.81574800:1130.3556800.3684880.82256000:1240.3629780.3677550.82351100:12| [0m 86      [0m | [0m 0.8235  [0m | [0m 0.06921 [0m | [0m 5.7     [0m | [0m 2.778   [0m |epochtrain_lossvalid_lossaccuracytime00.4485720.4512550.78976600:1010.4170930.4118380.80814300:1120.4007990.4001850.81622300:1230.3786410.3850820.82034200:1240.3652780.3803200.81844100:11| [0m 87      [0m | [0m 0.8184  [0m | [0m 7.965   [0m | [0m 5.261   [0m | [0m 2.661   [0m |epochtrain_lossvalid_lossaccuracytime00.3994530.4292880.78311200:0710.3689870.3769850.82572900:0720.3582260.3721030.83016500:0730.3489400.3620690.83111500:0740.3414370.3614700.83048200:07| [0m 88      [0m | [0m 0.8305  [0m | [0m 2.792   [0m | [0m 7.917   [0m | [0m 0.761   [0m |
| [0m 89      [0m | [0m 0.8294  [0m | [0m 7.995   [0m | [0m 7.186   [0m | [0m 0.1199  [0m |epochtrain_lossvalid_lossaccuracytime00.3909140.4037800.79990500:0710.3752410.4004060.82161000:0720.3597100.3732330.82604600:0730.3511080.3672550.82398600:0740.3482300.3628270.83048200:07| [0m 90      [0m | [0m 0.8305  [0m | [0m 2.526   [0m | [0m 3.741   [0m | [0m 0.1186  [0m |
| [0m 91      [0m | [0m 0.8294  [0m | [0m 0.03285 [0m | [0m 5.742   [0m | [0m 0.9747  [0m |epochtrain_lossvalid_lossaccuracytime00.3844850.4137110.81685700:0610.3583830.3821250.81305500:0620.3496320.3806440.82557000:0730.3500050.3638570.83349200:0740.3413420.3625330.82984800:07| [0m 92      [0m | [0m 0.8298  [0m | [0m 4.112e-0[0m | [0m 8.0     [0m | [0m 0.0     [0m |epochtrain_lossvalid_lossaccuracytime00.4618060.4543140.75221800:1110.4294290.4527210.76806100:1320.4087920.4223150.78501300:1230.4002900.4111340.80529200:1240.3969740.4093370.80465800:12| [0m 93      [0m | [0m 0.8047  [0m | [0m 4.563   [0m | [0m 0.6868  [0m | [0m 2.461   [0m |epochtrain_lossvalid_lossaccuracytime00.4649200.4402140.77265500:1110.4016110.4002400.80576700:1120.3919050.4145120.79863800:1330.3915610.4072710.80560800:1340.3875960.3975860.81463900:13| [0m 94      [0m | [0m 0.8146  [0m | [0m 4.697   [0m | [0m 3.412   [0m | [0m 2.514   [0m |epochtrain_lossvalid_lossaccuracytime00.4179170.4235880.75507000:0510.3901580.3969160.81764900:0720.3745620.3897640.82842200:0730.3683170.3852680.82271900:0740.3733670.3850920.82747100:07| [0m 95      [0m | [0m 0.8275  [0m | [0m 5.04    [0m | [0m 0.4492  [0m | [0m 0.5899  [0m |epochtrain_lossvalid_lossaccuracytime00.3689910.3882510.82240200:0710.3631110.3996700.81479700:0820.3703900.3777260.81844100:1030.3534930.3686030.82382800:1040.3554930.3682810.82366900:09| [0m 96      [0m | [0m 0.8237  [0m | [0m 0.6025  [0m | [0m 2.712   [0m | [0m 1.166   [0m |epochtrain_lossvalid_lossaccuracytime00.4261610.4175670.79372600:0710.3976330.3928520.81812400:0820.3710730.3716780.82176800:0930.3517160.3616070.83174900:1040.3472910.3598810.83206600:09| [95m 97      [0m | [95m 0.8321  [0m | [95m 6.389   [0m | [95m 3.648   [0m | [95m 1.016   [0m |epochtrain_lossvalid_lossaccuracytime00.4089370.4407550.78580500:1210.3698000.3796220.81986700:1220.3560480.3701190.82763000:1130.3542710.3660910.82636200:1140.3593240.3660200.82731300:12| [0m 98      [0m | [0m 0.8273  [0m | [0m 0.5927  [0m | [0m 1.715   [0m | [0m 2.847   [0m |epochtrain_lossvalid_lossaccuracytime00.4438310.4697750.75380200:1110.4116360.4233620.78786400:1120.4116690.4218740.79752900:1330.3922420.3953770.81638200:1240.3839470.3902700.81844100:12| [0m 99      [0m | [0m 0.8184  [0m | [0m 6.602   [0m | [0m 2.266   [0m | [0m 2.535   [0m |epochtrain_lossvalid_lossaccuracytime00.4136880.4691930.81115300:0810.3839080.4015200.81558900:0820.3639080.3751780.82493700:0830.3606940.3665360.82873900:0840.3418510.3628860.83016500:08| [0m 100     [0m | [0m 0.8302  [0m | [0m 3.099   [0m | [0m 6.058   [0m | [0m 1.058   [0m |epochtrain_lossvalid_lossaccuracytime00.4043240.4023100.80766800:0910.3836280.4281260.78057700:0920.3609170.3751980.82699600:0930.3499220.3671140.82826400:0940.3387150.3637620.83016500:09| [0m 101     [0m | [0m 0.8302  [0m | [0m 4.268   [0m | [0m 5.28    [0m | [0m 1.397   [0m |epochtrain_lossvalid_lossaccuracytime00.3813170.4105460.81543100:0610.3556320.4245770.80244000:0620.3611930.3776750.81812400:0730.3557510.3632050.82731300:0740.3388020.3624030.82747100:07| [0m 102     [0m | [0m 0.8275  [0m | [0m 4.735   [0m | [0m 3.398   [0m | [0m 0.01904 [0m |epochtrain_lossvalid_lossaccuracytime00.3811500.3984990.81321300:0710.3709970.4130400.79816200:0720.3641540.3690660.82335200:0730.3544340.3629250.82509500:0740.3487360.3621250.82446100:07| [0m 103     [0m | [0m 0.8245  [0m | [0m 0.0     [0m | [0m 6.371   [0m | [0m 9.721e-0[0m |epochtrain_lossvalid_lossaccuracytime00.3883780.4128390.81036100:0610.3775330.3804320.82256000:0620.3562420.3722500.82493700:0730.3462740.3642360.83032300:0740.3492730.3625970.83048200:07| [0m 104     [0m | [0m 0.8305  [0m | [0m 1.683   [0m | [0m 6.381   [0m | [0m 0.8564  [0m |epochtrain_lossvalid_lossaccuracytime00.4102860.4135060.80164800:0710.3867870.3973280.81226200:0720.3652680.3904190.82557000:0730.3685660.3869550.82826400:0740.3627510.3831240.83048200:07| [0m 105     [0m | [0m 0.8305  [0m | [0m 6.387   [0m | [0m 2.333   [0m | [0m 0.4877  [0m |epochtrain_lossvalid_lossaccuracytime00.4323470.4500280.78089400:1710.4147660.4025650.80956900:1720.3824950.3822810.81242100:1830.3668520.3731580.82208500:1840.3536920.3684710.82446100:18| [0m 106     [0m | [0m 0.8245  [0m | [0m 0.9452  [0m | [0m 6.902   [0m | [0m 3.991   [0m |epochtrain_lossvalid_lossaccuracytime00.3976040.4079660.81479700:0610.3867550.3826980.80006300:0720.3616720.3736100.82319400:0730.3456550.3632780.82984800:0740.3499310.3621710.83095700:07| [0m 107     [0m | [0m 0.831   [0m | [0m 5.475   [0m | [0m 5.721   [0m | [0m 0.02659 [0m |
| [0m 108     [0m | [0m 0.8308  [0m | [0m 4.597   [0m | [0m 7.994   [0m | [0m 0.1318  [0m |epochtrain_lossvalid_lossaccuracytime00.3917200.3951610.81955000:0610.3728590.3878590.81020300:0720.3673850.3769520.81226200:0730.3468100.3653120.82715500:0740.3412210.3635280.82905600:07| [0m 109     [0m | [0m 0.8291  [0m | [0m 2.391   [0m | [0m 4.915   [0m | [0m 0.8695  [0m |epochtrain_lossvalid_lossaccuracytime00.3916350.4021380.80465800:0810.4024250.4680130.80386600:0920.3684430.3729270.82366900:0930.3615790.3649130.82667900:0940.3479890.3634160.82858000:09| [0m 110     [0m | [0m 0.8286  [0m | [0m 5.711   [0m | [0m 7.031   [0m | [0m 1.157   [0m |epochtrain_lossvalid_lossaccuracytime00.3932250.4144610.81273800:0710.3776410.3819100.82065900:0720.3639790.3750420.82683800:0730.3498670.3635470.83016500:0740.3500350.3642540.83000600:07| [0m 111     [0m | [0m 0.83    [0m | [0m 3.867   [0m | [0m 7.351   [0m | [0m 0.7373  [0m |
| [0m 112     [0m | [0m 0.8275  [0m | [0m 5.568   [0m | [0m 0.8565  [0m | [0m 0.9522  [0m |
| [0m 113     [0m | [0m 0.8311  [0m | [0m 5.553   [0m | [0m 1.45    [0m | [0m 0.0     [0m |epochtrain_lossvalid_lossaccuracytime00.4042150.4212930.81780700:1010.3807530.3885330.80307400:0920.3624750.3811990.82034200:0930.3534160.3684310.82462000:0940.3358540.3654480.82557000:09| [0m 114     [0m | [0m 0.8256  [0m | [0m 3.949   [0m | [0m 1.125   [0m | [0m 1.109   [0m |epochtrain_lossvalid_lossaccuracytime00.3973510.4828810.76616000:0810.3731900.3729990.81907500:0820.3540750.3684550.82604600:0830.3506170.3627930.83032300:0840.3473810.3619800.83127400:08| [0m 115     [0m | [0m 0.8313  [0m | [0m 7.258   [0m | [0m 4.724   [0m | [0m 1.639   [0m |
=============================================================optimizer.max['target']0.8428390622138977{key:2**int(value)forkey,valueinoptimizer.max['params'].items()}{'pow_n_a': 2, 'pow_n_d': 64, 'pow_n_steps': 1}Out of memory datasetIf your dataset is so big it doesn't fit in memory, you can load a chunk of it each epoch.df=pd.read_csv(path/'adult.csv')df_main,df_valid=df.iloc[:-1000].copy(),df.iloc[-1000:].copy()# choose size that fit in memorydataset_size=1000# load chunk with your own codedefload_chunk():returndf_main.sample(dataset_size).copy()df_small=load_chunk()cat_names=['workclass','education','marital-status','occupation','relationship','race','native-country','sex']cont_names=['age','fnlwgt','education-num']procs=[Categorify,FillMissing,Normalize]splits=RandomSplitter()(range_of(df_small))to=TabularPandas(df_small,procs,cat_names,cont_names,y_names="salary",y_block=CategoryBlock(),splits=None,do_setup=True)# save the validation setto_valid=to.new(df_valid)to_valid.process()val_dl=TabDataLoader(to_valid.train)len(to.train)1000classReloadCallback(Callback):defbegin_epoch(self):df_small=load_chunk()to_new=to.new(df_small)to_new.process()trn_dl=TabDataLoader(to_new.train)self.learn.dls=DataLoaders(trn_dl,val_dl).cuda()dls=to.dataloaders()emb_szs=get_emb_sz(to)model=TabNetModel(emb_szs,len(to.cont_names),dls.c,n_d=8,n_a=32,n_steps=1);opt_func=partial(Adam,wd=0.01,eps=1e-5)learn=Learner(dls,model,CrossEntropyLossFlat(),opt_func=opt_func,lr=3e-2,metrics=[accuracy])learn.add_cb(ReloadCallback());learn.fit_one_cycle(10)epochtrain_lossvalid_lossaccuracytime00.5877400.5505440.75600000:0110.5454110.5157720.78200000:0120.4842890.4685860.81300000:0130.4471110.4357740.81700000:0140.4490500.3947150.81900000:0150.4288630.3820050.83500000:0160.3821000.4042580.82600000:0170.3839150.3761790.83300000:0180.3894600.3678570.83400000:0190.3764860.3675770.83400000:01