Named Entity Recognition in Ecommerce Industry  Custom model [Github Repo]  03/07/24
Source: tai_blog

Github Repo: https://github.com/vaibhawkhemka/ML-Umbrella/tree/main/NLP/Product-Categorization   From e-commerce to Customer support  all businesses require some kind of NER model to process huge amounts of texts from users.   To automate this whole  one requires NER models to extract relevant and important entities from text.   Final Result/OutputInput text = EL D68 (Green  32 GB) 3 GB RAM [3 GB RAM U+007C 32 GB ROM U+007C Expandable Upto 128 GB  15.46 cm (6.088 inch) Display  13MP Rear Camera U+007C 8MP Front Camera  4000 mAh Battery  Quad-Core Processor]   Output =   Green ->>>> COLOR 32 GB ->>>> STORAGE 3 GB RAM ->>>> RAM 3 GB RAM ->>>> RAM 32 GB ROM ->>>> STORAGE Expandable Upto 128 GB ->>>> EXPANDABLE_STORAGE 15.46 cm (6.088 inch) ->>>> SCREEN_SIZE 13MP Rear Camera ->>>> BACK_CAMERA 8MP Front Camera ->>>> FRONT_CAMERA 4000 mAh Battery ->>>> BATTERY_CAPACITY Quad-Core Processor ->>>> PROCESSOR_CORE   Data PreparationA tool for creating this dataset (https://github.com/tecoholic/ner-annotator)    Snapshot for the dataset for Mobile phone product description on Amazon:   A single record of the Data:   Converting into proper Spacy span format:The proper format that Spacy Ner model understands   import jsonlines  json file_path = Training Data/Mobile/Mobile_training.jsonl laptop_classes = [RAM STORAGE BATTERY CAPACITY PROCESSOR_TYPE SCREEN_SIZE REFRESH_RATE SCREEN_TYPE BACK_CAMERA FRONT_CAMERA] with jsonlines.open(file_path) as reader: output_json = {classes: laptop_classes  annotations: []} # Iterate over each line (JSON object) for obj in reader: processed_obj = [obj[text] {entities:obj[label]}] output_json[annotations].append(processed_obj) # Save the output JSON to a new file with open('Training Data/Mobile/Mobile_annotations.json'  'w') as f: json.dump(output_json  f  indent=None)Above is the code for converting into proper data format. Check out jupyter notebook: NER_model_Mobile.ipynb   Final pandas dataframe from processed data:   Splitting the dataset  10% test### Split the data from sklearn.model_selection import train_test_split train  test = train_test_split(df  test_size=0.1) train.head()Create spacy DocBin objects from annotated data to train Spacy NER model:import spacy from spacy.tokens import DocBin from tqdm import tqdm # Define a function to create spaCy DocBin objects from the annotated data def get_spacy_doc(data): # Create a blank spaCy pipeline nlp = spacy.blank('en') db = DocBin() # Initialize a counter for None spans none_spans = 0 spans = 0 for index  row in data.iterrows(): # Get the text and annotations text = row[Description] annotations = row[Annotations] # Check if the text is not empty if not text: continue # Process the text and annotations doc = nlp(text) if doc is None: print(fFailed to process text: {text}) continue ents = [] for start  end  label in annotations: if start < 0 or end < 0: print(fInvalid annotation: {start}  {end}  {label}) continue #print(text) span = doc.char_span(start  end  label=label) if span is None: print(fFailed to create span for annotation: {start}  {end}  {label}) none_spans += 1 continue else: spans+=1 ents.append(span) doc.ents = ents #Add the processed document to the DocBin db.add(doc) print(fNumber of None spans: {none_spans}) print(fNumber of spans: {spans}) return dbModellingArchitecture:The basic architecture for all spacy models:   Reference: https://explosion.ai/blog/deep-learning-formula-nlp   [Embed]HashEmbed  Sub-word features than character based richer representation and arbitrary sized vocabulary  Can use Word2vec/Glove etc   [Encode]  Context-independent to context-dependent using LSTM or CNN.   [Attend]  Attention mechanism by Key  Value pair  and context vectors   [Predict]  MLP   Tok2vec model [example]:   https://github.com/explosion/spaCy/blob/master/spacy/ml/models/tok2vec.py (Built using thinc framework)   NER Model  Transition-Based:   State(all three stack  buffer  and output) and Action   Structure Prediction.   The above shows how the transition-based approach works with stack  buffer  output  and Transition/action.   Reference: https://www.microsoft.com/en-us/research/video/transition-based-natural-language-processing/   The above shows How stacked LSTM works for encoding for all states and actions.   The final Prediction from MLP is the Multiclassification task with labels as SHIFT  OUT  and REDUCE   Spacy model layer and Config Mapping:   Example of a tok2vec config:   Model in thinc framework:   Respective config for the model:   Thinc deep learning framework is used as a backend to build spacy models instead of pytorch or TensorFlow.   Difference between normal pytorch and spacy models. => Spacy(easy  reliable and productionable)   The user can define and create this model using a configuration file for any task: NER  Tok2Vec  Tagger  Dependency Parser  Sentiment etc   One can also create thinc models and wrap around pytorch and TensorFlow. I will build it next blog.   NER Config file created here:   Reference: https://spacy.io/usage/training   config_ner.cfg :   [paths] train = null dev = null vectors = en_core_web_lg init_tok2vec = null [system] gpu_allocator = null seed = 0 [nlp] lang = en pipeline = [tok2vec ner] batch_size = 1000 disabled = [] before_creation = null after_creation = null after_pipeline_creation = null tokenizer = {@tokenizers:spacy.Tokenizer.v1} vectors = {@vectors:spacy.Vectors.v1} [components] [components.ner] factory = ner incorrect_spans_key = null moves = null scorer = {@scorers:spacy.ner_scorer.v1} update_with_oracle_cut_size = 100 [components.ner.model] @architectures = spacy.TransitionBasedParser.v2 state_type = ner extra_state_tokens = false hidden_width = 64 maxout_pieces = 2 use_upper = true nO = null [components.ner.model.tok2vec] @architectures = spacy.Tok2VecListener.v1 width = ${components.tok2vec.model.encode.width} upstream = * [components.tok2vec] factory = tok2vec [components.tok2vec.model] @architectures = spacy.Tok2Vec.v2 [components.tok2vec.model.embed] @architectures = spacy.MultiHashEmbed.v2 width = ${components.tok2vec.model.encode.width} attrs = [NORM PREFIX SUFFIX SHAPE] rows = [5000 1000 2500 2500] include_static_vectors = true [components.tok2vec.model.encode] @architectures = spacy.MaxoutWindowEncoder.v2 width = 256 depth = 8 window_size = 1 maxout_pieces = 3 [corpora] [corpora.dev] @readers = spacy.Corpus.v1 path = ${paths.dev} max_length = 0 gold_preproc = false limit = 0 augmenter = null [corpora.train] @readers = spacy.Corpus.v1 path = ${paths.train} max_length = 0 gold_preproc = false limit = 0 augmenter = null [training] dev_corpus = corpora.dev train_corpus = corpora.train seed = ${system.seed} gpu_allocator = ${system.gpu_allocator} dropout = 0.1 accumulate_gradient = 1 patience = 1600 max_epochs = 0 max_steps = 20000 eval_frequency = 200 frozen_components = [] annotating_components = [] before_to_disk = null before_update = null [training.batcher] @batchers = spacy.batch_by_words.v1 discard_oversize = false tolerance = 0.2 get_length = null [training.batcher.size] @schedules = compounding.v1 start = 100 stop = 1000 compound = 1.001 t = 0.0 [training.logger] @loggers = spacy.ConsoleLogger.v1 progress_bar = false [training.optimizer] @optimizers = Adam.v1 beta1 = 0.9 beta2 = 0.999 L2_is_weight_decay = true L2 = 0.01 grad_clip = 1.0 use_averages = false eps = 0.00000001 learn_rate = 0.001 [training.score_weights] ents_f = 1.0 ents_p = 0.0 ents_r = 0.0 ents_per_type = null [pretraining] [initialize] vectors = ${paths.vectors} init_tok2vec = ${paths.init_tok2vec} vocab_data = null lookups = null before_init = null after_init = null [initialize.components] [initialize.tokenizer]Output and Evaluation:Evaluation is done based on ENTS_P(Precision)  ENTS_R(Recall) and ENTS_F (F-Score).   After the 15th epoch Final ENTS_F is 57.64  which can be improved by providing more data for this case.   Intuition for Evaluation:We evaluate the NER model based on Span-Identification and Span-Prediction.   Span-Identification:   https://cees-roele.medium.com/custom-evaluation-of-spans-in-spacy-f1f2e7a99ad8   As discussed  NER is a multiclass Classification problem with SHIFT  OUT  and REDUCE as output. But we evaluate our models only based on REDUCE.   The above picture shows how Precision  Recall  and F-Score are calculated.   The code used for evaluating PRF (Precision-Recall-Fscore) by spacy:   def get_ner_prf(examples: Iterable[Example]  **kwargs) -> Dict[str  Any]: Compute micro-PRF and per-entity PRF scores for a sequence of examples. score_per_type = defaultdict(PRFScore) for eg in examples: if not eg.y.has_annotation(ENT_IOB): continue golds = {(e.label_  e.start  e.end) for e in eg.y.ents} align_x2y = eg.alignment.x2y for pred_ent in eg.x.ents: if pred_ent.label_ not in score_per_type: score_per_type[pred_ent.label_] = PRFScore() indices = align_x2y[pred_ent.start : pred_ent.end] if len(indices): g_span = eg.y[indices[0] : indices[-1] + 1] # Check we aren't missing annotation on this span. If so  # our prediction is neither right nor wrong  we just # ignore it. if all(token.ent_iob != 0 for token in g_span): key = (pred_ent.label_  indices[0]  indices[-1] + 1) if key in golds: score_per_type[pred_ent.label_].tp += 1 golds.remove(key) else: score_per_type[pred_ent.label_].fp += 1 for label  start  end in golds: score_per_type[label].fn += 1 totals = PRFScore() for prf in score_per_type.values(): totals += prf if len(totals) > 0: return { ents_p: totals.precision  ents_r: totals.recall  ents_f: totals.fscore  ents_per_type: {k: v.to_dict() for k  v in score_per_type.items()}  } else: return { ents_p: None  ents_r: None  ents_f: None  ents_per_type: None  }Reference: https://github.com/explosion/spaCy/blob/master/spacy/scorer.py#L760   Span Prediction :   There are 9 different entires like [RAM  STORAGE  BATTERY CAPACITY  PROCESSOR_TYPE  SCREEN_SIZE  REFRESH_RATE  SCREEN_TYPE  BACK_CAMERA  FRONT_CAMERA] to predict for REDUCE class.   It uses categorical crossentropy loss function to optimize NER models (More details in later blogs)   Testing and Final Results:Input text = EL D68 (Green  32 GB) 3 GB RAM [3 GB RAM U+007C 32 GB ROM U+007C Expandable Upto 128 GB  15.46 cm (6.088 inch) Display  13MP Rear Camera U+007C 8MP Front Camera  4000 mAh Battery  Quad-Core Processor]   Output =   Green ->>>> COLOR 32 GB ->>>> STORAGE 3 GB RAM ->>>> RAM 3 GB RAM ->>>> RAM 32 GB ROM ->>>> STORAGE Expandable Upto 128 GB ->>>> EXPANDABLE_STORAGE 15.46 cm (6.088 inch) ->>>> SCREEN_SIZE 13MP Rear Camera ->>>> BACK_CAMERA 8MP Front Camera ->>>> FRONT_CAMERA 4000 mAh Battery ->>>> BATTERY_CAPACITY Quad-Core Processor ->>>> PROCESSOR_CORE   Github Link: https://github.com/vaibhawkhemka/ML-Umbrella/tree/main/NLP/Product-Categorization   Thanks for reading the blog.   If you have any questions  hit me up on my LinkedIn: https://www.linkedin.com/in/vaibhaw-khemka-a92156176/   References for modeling:   https://explosion.ai/blog/deep-learning-formula-nlp => Embed  Encode  Attend and Predict => Position is imp in sequence in text.   https://support.prodi.gy/t/spacy-ner-models-architecture-details/4336   https://github.com/explosion/spaCy/blob/master/spacy/ml/models/tok2vec.py   https://spacy.io/usage/layers-architectures   https://spacy.io/api/architectures#CharacterEmbed   Understanding span:   https://spacy.io/api/span