NLP | IOB tags - GeeksforGeeks

16 Jul, 2021
What are Chunks ? Chunks are made up of words and the kinds of words are defined using the part-of-speech tags. One can even define a pattern or words that can’t be a part of chuck and such words are known as chinks.  
What are IOB tags ? It is a format for chunks. These tags are similar to part-of-speech tags but can denote the inside, outside, and beginning of a chunk. Not just noun phrase but multiple different chunk phrase types are allowed here. Example : It is an excerpt from the conll2000 corpus. Each word is with a part-of-speech tag followed by an IOB tag on its own line: 
Mr. NNP B-NP
Meador NNP I-NP
had VBD B-VP
been VBN I-VP
executive JJ B-NP
vice NN I-NP
president NN I-NP
of IN B-PP
Balcor NNP B-NP
What it means ? B-NP : beginning of a noun phrase I-NP : describes that the word is inside of the current noun phrase. O : end of the sentence. B-VP and I-VP : beginning and inside of a verb phrase.Code #1 : How it works – chunking words with IOB tags.  
Python3
# Loading the librariesfrom nltk.corpus.reader import ConllChunkCorpusReader # Initializingreader = ConllChunkCorpusReader(        '.', r'.*\.iob', ('NP', 'VP', 'PP')) reader.chunked_words() reader.iob_words()
Output :  
[Tree('NP', [('Mr.', 'NNP'), ('Meador', 'NNP')]), Tree('VP', [('had', 'VBD'), 
('been', 'VBN')]), ...]

[('Mr.', 'NNP', 'B-NP'), ('Meador', 'NNP', 'I-NP'), ...]
Code #2 : How it works – chunking sentence with IOB tags.  
Python3
# Loading the librariesfrom nltk.corpus.reader import ConllChunkCorpusReader # Initializingreader = ConllChunkCorpusReader(        '.', r'.*\.iob', ('NP', 'VP', 'PP')) reader.chunked_sents() reader.iob_sents()
Output :  
[Tree('S', [Tree('NP', [('Mr.', 'NNP'), ('Meador', 'NNP')]),
Tree('VP', [('had', 'VBD'), ('been', 'VBN')]), 
Tree('NP', [('executive', 'JJ'), ('vice', 'NN'), ('president', 'NN')]),
Tree('PP', [('of', 'IN')]), Tree('NP', [('Balcor', 'NNP')]), ('.', '.')])]

[[('Mr.', 'NNP', 'B-NP'), ('Meador', 'NNP', 'I-NP'), ('had', 'VBD', 'B-VP'), 
('been', 'VBN', 'I-VP'), ('executive', 'JJ', 'B-NP'), ('vice', 'NN', 'I-NP'), 
('president', 'NN', 'I-NP'), ('of', 'IN', 'B-PP'), ('Balcor', 'NNP', 'B-NP'), 
('.', '.', 'O')]]
Let’s understand the code above :  
For reading the corpus with IOB format, ConllChunkCorpusReader class is used.
No separation of paragraphs and each sentence is separated by a blank line, therefore para_* methods are not available.
Tuple or list specifying the types of chunks in the file like (‘NP’, ‘VP’, ‘PP’) sreves as the third argument to ConllChunkCorpusReader.
iob_words() and iob_sents() methods returns lists of three tuples of (word, pos, iob)
Code #3 : Tree Leaves – i.e. the tagged tokens  
Python3
# Loading the librariesfrom nltk.corpus.reader import ConllChunkCorpusReader # Initializingreader = ConllChunkCorpusReader(        '.', r'.*\.iob', ('NP', 'VP', 'PP')) reader.chunked_words()[0].leaves() reader.chunked_sents()[0].leaves() reader.chunked_paras()[0][0].leaves()
Output :  
[('Earlier', 'JJR'), ('staff-reduction', 'NN'), ('moves', 'NNS')]

[('Earlier', 'JJR'), ('staff-reduction', 'NN'), ('moves', 'NNS'),
('have', 'VBP'), ('trimmed', 'VBN'), ('about', 'IN'), ('300', 'CD'),
('jobs', 'NNS'), (', ', ', '), ('the', 'DT'), ('spokesman', 'NN'),
('said', 'VBD'), ('.', '.')]

[('Earlier', 'JJR'), ('staff-reduction', 'NN'), ('moves', 'NNS'),
('have', 'VBP'), ('trimmed', 'VBN'), ('about', 'IN'), ('300', 'CD'),
('jobs', 'NNS'), (', ', ', '), ('the', 'DT'), ('spokesman', 'NN'),
('said', 'VBD'), ('.', '.')]
 
Akanksha_Rai
simmytarika5
saurabh1990aror
mrtarunbhatia
Natural-language-processing
Python-nltk
Machine Learning
Python
Machine Learning
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
ML | Linear Regression
Reinforcement learning
Decision Tree
Activation functions in Neural Networks
Decision Tree Introduction with example
Read JSON file using Python
Adding new column to existing DataFrame in Pandas
Python map() function
How to get column names in Pandas dataframe