Package: dnnnlp
PyTorch - Deep Neural Network - Natural Language ProcessingVersion 1.1 by KzXuanContains CNN, RNN and Transformer layers and models implemented by pytorch for classification and sequence labeling tasks in NLP.Newly designed modules.Reduce usage complexity.Usemaskas the sequence length identifier.Multi-GPU parallel for grid search.Dependeciespython 3.5+ & pytorch 1.2.0+Install>pipinstalldnnnlpAPI Document(In Chinese)Layer-layer.pyModel-model.pyExecution-exec.pyUtility-utils.pyHyperparametersNameTypeDefaultDescriptionn_gpuint1The number of GPUs (0 means no GPU acceleration).space_turboboolTrueAccelerate with more GPU memories.rand_seedint100Random seed setting.data_shuffleboolTureDisrupt data for training.emb_typestrNoneEmbedding modes contain None, 'const' or 'variable'.emb_dimint300Embedding dimension (or feature dimension).n_classint2Number of target classes.n_hiddenint50Number of hidden nodes, or output channels of CNN.learning_ratefloat0.01Learning rate.l2_regfloat1e-6L2 regular.batch_sizeint32Number of samples for one batch.iter_timesint30Number of iterations.display_stepint2The number of iterations between each output of the result.drop_probfloat0.1Dropout ratio.eval_metricstr'accuracy'Evaluation metrics contain 'accuracy', 'macro', 'class1', etc.Usage# import our modulesfromdnnnlp.modelimportRNNModelfromdnnnlp.execimportdefault_args,Classify# load the embedding matrixemb_mat=np.array((-1,300))# load the train datatrain_x=np.array((800,50))train_y=np.array((800,))train_mask=np.array((800,50))# load the test datatest_x=np.array((200,50))test_y=np.array((200,))test_mask=np.array((200,50))# get the default argumentsargs=default_args()# modify part of the argumentsargs.space_turbo=Falseargs.n_hidden=100args.batch_size=32Classification# initilize a modelmodel=RNNModel(args,emb_mat,bi_direction=False,rnn_type='GRU',use_attention=True)# initilize a classifiernn=Classify(model,args,train_x,train_y,train_mask,test_x,test_y,test_mask)# do training and testingevals=nn.train_test(device_id=0)Run several times and get the average score.# initilize a modelmodel=CNNModel(args,emb_mat,kernel_widths=[2,3,4])# initilize a classifiernn=Classify(model,args,train_x,train_y,train_mask)# run the model several timesavg_evals=average_several_run(nn.train_test,args,n_times=8,n_paral=4,fold=5)Parameters' grid search.# initilize a modelmodel=TransformerModel(args,n_layer=12,n_head=8)# initilize a classifiernn=Classify(model,args,train_x,train_y,train_mask,test_x,test_y,test_mask)# set searching paramsparams_search={'learning_rate':[0.1,0.01],'n_hidden':[50,100]}# run grid searchmax_evals=grid_search(nn,nn.train_test,args,params_search)Sequence labelingfromdnnnlp.modelimportRNNCRFModelfromdnnnlp.execimportdefault_args,SequenceLabeling# load the train datatrain_x=np.array((1000,50))train_y=np.array((1000,50))train_mask=np.array((1000,50))# initilize a modelmodel=RNNCRFModel(args)# initilize a labelernn=SequenceLabeling(model,args,train_x,train_y,train_mask)# do cross validationnn.cross_validation(fold=5)Historyversion 1.1AddCRFLayer: packaging CRF for both training and testing.AddRNNCRFModel: a integrated RNN-CRF sequence labeling model.AddSequenceLabeling: a sequence labeling execution module that inherits from Classify.Fix errors in judging whether a tensor is None.version 1.0Rename projectdnntodnnnlp.Remove filebase, add fileutils.Optimize and renameSoftmaxLayerandSoftAttentionLayer.Rewrite and renameEmbeddingLayer,CNNLayerandRNNLayer.RewriteMultiheadAttentionLayer: a packaging attention layer based onnn.MultiheadAttention.RewriteTransformerLayer: support newMultiheadAttentionLayer.Optimize and renameCNNModel,RNNModelandTransformerModel.Optimize and renameClassify: a highly applicable classification execution module.Rewriteaverage_several_runandgrid_search: support multi-GPU parallel.Support pytorch 1.2.0.version 0.12UpdateRNN_layer: fully support for tanh, LSTM and GRU.Fix errors in some mask operations.Support pytorch 1.1.0.Old version0.12.3.version 0.11Provides an acceleration method by using more GPU memories.Fix the problem of memory consumption caused by abnormal data reading.Addmulti_head_attention_layer: packaging multi-head attention for Transformer.AddTransformer_layerandTransformer_model: packaging Transformer layer and model written by ourself.Support data disruption for training.version 0.10Split the code into four files:base,layer,model,exec.AddCNN_layerandCNN_model: packaging CNN layer and model.Support multi-GPU parallel for each model.version 0.9Fix the problem of output format.Fix the statistical errors in cross-validation part ofLSTM_classify.Rename:LSTM_modeltoRNN_layer,self_attentiontoself_attention_layer.Addsoftmax_layer: a packaging fully-connected layer.version 0.8Adjust the applicability of functions inLSTM_classifyto avoid rewriting inLSTM_sequence.Optimize the way of parameter transfer.A more complete evaluation mechanism.version 0.7AddLSTM_sequence: a sequence labeling module forLSTM_model.Fix the nan-value problem in hierarchical classification.Support pytorch 1.0.0.version 0.6UpdateLSTM_classify: support hierarchical classification.TheGRU_modelis merged into theLSTM_model.Adapt to CPU operation.version 0.5Split the running part ofLSTM_classifyto reduce the rewrite of custom models.Add control for visual output.Create functionaverage_several_run: support to get the average score after several training and testing.Create functiongrid_search: support parameters' grid search.version 0.4AddGRU_model: a packaging GRU model based onnn.GRU.Support L2 regular.version 0.3Addself_attention: provides attention mechanism support.UpdateLSTM_classify: adapts to complex custom models.version 0.2Support mode selection of embedding.Default usage ofnn.Dropout.Create functiondefault_argsto provide default hyperparameters.version 0.1Initilization of projectdnn: based on pytorch 0.4.1.AddLSTM_model: a packaging LSTM model based onnn.LSTM.AddLSTM_classify: a classification module for LSTM model, which supports train-test and corss-validation.