antu Documentation
Release 0.0.1

AntNLPMay 24, 2020

Contents:

1

antu.io package
1.1 Subpackages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Submodules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3 antu.io.instance module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.4 antu.io.vocabulary module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.5 Module contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3
3
9
9
11
13

2

antu.nn package
2.1 Subpackages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Module contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15
15
15

3

Indices and tables

17

Python Module Index

19

Index

21

i

ii

antu Documentation, Release 0.0.1

Universal data IO and neural network modules in NLP tasks.
• *data IO is an universal module in Natural Language Processing system and not based on any framework (likeTensorFlow, PyTorch, MXNet, Dynet. . . ).
• *neural network module contains the neural network structures commonly used in NLP tasks. We want to designcommonly used structures for each neural network framework. We will continue to develop this module.

Contents:1

antu Documentation, Release 0.0.1

2

Contents:

CHAPTER

1

antu.io package

1.1 Subpackages
1.1.1 antu.io.dataset_readers package
Submodules
antu.io.dataset_readers.dataset_reader moduleclass antu.io.dataset_readers.dataset_reader.DatasetReader
Bases: object
Methods
input_to_instance
read
input_to_instance(inputs: str) → antu.io.instance.Instanceread(file_path: str) → List[antu.io.instance.Instance]
Module contents

1.1.2 antu.io.datasets package
Submodules

3antu Documentation, Release 0.0.1

antu.io.datasets.dataset moduleclass antu.io.datasets.dataset.Dataset
Bases: object
Methods
build_dataset
build_dataset()
datasets = {}vocabulary_set = {}Module contents

1.1.3 antu.io.fields package
Submodules
antu.io.fields.field moduleclass antu.io.fields.field.Field
Bases: object
A Field is an ingredient of a data instance. In most NLP tasks, Field stores data of string types. It contains
one or more indexers that map string data to the corresponding index. Data instances are collections of fields.
Methods
count_vocab_items(counter, Dict[str, int]])
index(vocab)

We count the number of strings if the string needs tobe mapped to one or more integers.
Gets one or more index mappings for each elementin the Field.

count_vocab_items(counter: Dict[str, Dict[str, int]]) → None
We count the number of strings if the string needs to be mapped to one or more integers. You can passdirectly if there is no string that needs to be mapped.
Parameterscounter [Dict[str, Dict[str, int]]]‘‘counter‘‘ is used to count the number of each item. The first key
represents the namespace of the vocabulary, and the second key represents
the string of the item.
index(vocab: antu.io.vocabulary.Vocabulary) → None
Gets one or more index mappings for each element in the Field.
Parameters

4

Chapter 1. antu.io package

antu Documentation, Release 0.0.1

vocab [Vocabulary]‘‘vocab‘‘ is used to get the index of each item.
antu.io.fields.index_field moduleclass antu.io.fields.index_field.IndexField(name: str, tokens: List[str])
Bases: antu.io.fields.field.Field
A IndexField is an integer field, and we can use it to store data ID.
Parametersname [str] Field name. This is necessary and must be unique (not the same as other fieldnames).
tokens [List[str]] Field content that contains a list of string.
Methods
count_vocab_items(counters, Dict[str, int]])
index(vocab)

IndexField doesn’t need index operation.
IndexField doesn’t need index operation.

count_vocab_items(counters: Dict[str, Dict[str, int]]) → None
IndexField doesn’t need index operation.
index(vocab: antu.io.vocabulary.Vocabulary) → None
IndexField doesn’t need index operation.
antu.io.fields.sequence_label_field moduleclass antu.io.fields.sequence_label_field.SequenceLabelField(name: str, tokens:
List[str], indexers:
List[antu.io.token_indexers.token_indexer.To
Bases: antu.io.fields.field.Field
Methods
count_vocab_items(counters, Dict[str, int]])
index(vocab)

We count the number of strings if the string needs tobe mapped to one or more integers.
Gets one or more index mappings for each elementin the Field.

count_vocab_items(counters: Dict[str, Dict[str, int]]) → None
We count the number of strings if the string needs to be mapped to one or more integers. You can passdirectly if there is no string that needs to be mapped.
Parameterscounter [Dict[str, Dict[str, int]]]‘‘counter‘‘ is used to count the number of each item. The first key
represents the namespace of the vocabulary, and the second key represents

1.1. Subpackages

5antu Documentation, Release 0.0.1

the string of the item.
index(vocab: antu.io.vocabulary.Vocabulary) → None
Gets one or more index mappings for each element in the Field.
Parameters
vocab [Vocabulary]‘‘vocab‘‘ is used to get the index of each item.
antu.io.fields.text_field moduleclass antu.io.fields.text_field.TextField(name:str, tokens:
List[str], indexers:
List[antu.io.token_indexers.token_indexer.TokenIndexer]= [])Bases: antu.io.fields.field.Field
A TextField is a data field that is commonly used in NLP tasks, and we can use it to store text sequencessuch as sentences, paragraphs, POS tags, and so on.
Parametersname [str] Field name. This is necessary and must be unique (not the same as other fieldnames).
tokens [List[str]] Field content that contains a list of string.
indexers [List[TokenIndexer], optional (default=‘‘list()‘‘)] Indexer list that defines the
vocabularies associated with the field.
Methods
count_vocab_items(counters, Dict[str, int]])
index(vocab)

We count the number of strings if the string needs tobe counted to someGets one or more index mappings for each elementin the Field.

count_vocab_items(counters: Dict[str, Dict[str, int]]) → None
We count the number of strings if the string needs to be counted to some counters. You can pass directly if there is no string that needs
to be counted.
Parameterscounters [Dict[str, Dict[str, int]]] Element statistics for datasets. if field indexers indicate that this field is related to some counters, we use field content to update
the counters.
index(vocab: antu.io.vocabulary.Vocabulary) → None
Gets one or more index mappings for each element in the Field.
Parameters
vocab [Vocabulary] vocab is used to get the index of each item.

6

Chapter 1. antu.io package

antu Documentation, Release 0.0.1

Module contents

1.1.4 antu.io.token_indexers package
Submodules
antu.io.token_indexers.char_token_indexer moduleclass antu.io.token_indexers.char_token_indexer.CharTokenIndexer(related_vocabs:
List[str],transform:
Callable[[str],str] = <function
CharTokenIndexer.<lambda>>)Bases: antu.io.token_indexers.token_indexer.TokenIndexer
A CharTokenIndexer determines how string token get represented as arrays of list of character indices in amodel.
Parameters
related_vocabs [List[str]] Which vocabularies are related to the indexer.
transform [Callable[[str,], str], optional (default=‘‘lambda x:x‘‘)] What changes
need to be made to the token when counting or indexing. Commonly used are lowercase
transformation functions.
Methods
count_vocab_items(token, counters, Dict[str,...)tokens_to_indices(tokens, vocab)

Each character in the token is counted directly as anelement.
Takes a list of tokens and converts them to one ormore sets of indices.

count_vocab_items(token: str, counters: Dict[str, Dict[str, int]]) → None
Each character in the token is counted directly as an element.
Parameterscounter [Dict[str, Dict[str, int]]] We count the number of strings if the stringneeds to be counted to some counters.
tokens_to_indices(tokens: List[str], vocab: antu.io.vocabulary.Vocabulary) → Dict[str,
List[List[int]]]
Takes a list of tokens and converts them to one or more sets of indices. During the indexing process, eachtoken item corresponds to a list of index in the vocabulary.
Parameters
vocab [Vocabulary] vocab is used to get the index of each item.

1.1. Subpackages

7antu Documentation, Release 0.0.1

antu.io.token_indexers.single_id_token_indexer moduleclass antu.io.token_indexers.single_id_token_indexer.SingleIdTokenIndexer(related_vocabs:
List[str],transform:
Callable[[str],
str]=<function
SingleIdTokenIndexer.<lambda>>)Bases: antu.io.token_indexers.token_indexer.TokenIndexer
A SingleIdTokenIndexer determines how string token get represented as arrays of single id indices in amodel.
Parameters
related_vocabs [List[str]] Which vocabularies are related to the indexer.
transform [Callable[[str,], str], optional (default=‘‘lambda x:x‘‘)] What changes
need to be made to the token when counting or indexing. Commonly used are lowercase
transformation functions.
Methods
count_vocab_items(token, counters, Dict[str,...)tokens_to_indices(tokens, vocab)

The token is counted directly as an element.
Takes a list of tokens and converts them to one ormore sets of indices.

count_vocab_items(token: str, counters: Dict[str, Dict[str, int]]) → None
The token is counted directly as an element.
Parameterscounter [Dict[str, Dict[str, int]]] We count the number of strings if the stringneeds to be counted to some counters.
tokens_to_indices(tokens: List[str], vocab: antu.io.vocabulary.Vocabulary) → Dict[str, List[int]]
Takes a list of tokens and converts them to one or more sets of indices. During the indexing process, each
item corresponds to an index in the vocabulary.
Parameters
vocab [Vocabulary] vocab is used to get the index of each item.
Returns
res [Dict[str, List[int]]] if the token and index list is [w1:5, w2:3, w3:0], theresult will be {‘vocab_name’ : [5, 3, 0]}8Chapter 1. antu.io package

antu Documentation, Release 0.0.1

antu.io.token_indexers.token_indexer moduleclass antu.io.token_indexers.token_indexer.TokenIndexer
Bases: object
A TokenIndexer determines how string tokens get represented as arrays of indices in a model.
Methods
count_vocab_items(token, counter, Dict[str,...)tokens_to_indices(tokens, vocab)

Defines how each token in the field is counted.
Takes a list of tokens and converts them to one ormore sets of indices.

count_vocab_items(token: str, counter: Dict[str, Dict[str, int]]) → None
Defines how each token in the field is counted. In most cases, just use the string as a key. However, forcharacter-level TokenIndexer, you need to traverse each character in the string.
Parameterscounter [Dict[str, Dict[str, int]]] We count the number of strings if the stringneeds to be counted to some counters.
tokens_to_indices(tokens: List[str], vocab: antu.io.vocabulary.Vocabulary) → Dict[str, Indices]
Takes a list of tokens and converts them to one or more sets of indices. This could be just an ID for eachtoken from the vocabulary.
Parameters
vocab [Vocabulary] vocab is used to get the index of each item.
Module contents

1.2 Submodules
1.3 antu.io.instance moduleclass antu.io.instance.Instance(fields: List[antu.io.fields.field.Field] = None)
Bases: collections.abc.Mapping, typing.Generic
An Instance is a collection (list) of multiple data fields.
Parameters
fields [List[Field], optional (default=‘‘None‘‘)] A list of multiple data fields.
Methods
add_field(field)
count_vocab_items(counter, Dict[str, int]])

1.2. Submodules

Add the field to the existing Instance.
Increments counts in the given counter for all of
the vocabulary items in all of the Fields in thisInstance.
Continued on next page9

antu Documentation, Release 0.0.1

Table 8 – continued from previous page
dynamic_index_fields(vocab,
dy- Indexes all fields in this Instance using the pronamic_fields)
vided Vocabulary.
get(k[,d])index_fields(vocab)
Indexes all fields in this Instance using the provided Vocabulary.
items()
keys()
values()add_field(field: antu.io.fields.field.Field) → None
Add the field to the existing Instance.
Parameters
field [Field] Which field needs to be added.
count_vocab_items(counter: Dict[str, Dict[str, int]]) → None
Increments counts in the given counter for all of the vocabulary items in all of the Fields in thisInstance.
Parameterscounter [Dict[str, Dict[str, int]]] We count the number of strings if the stringneeds to be counted to some counters.
dynamic_index_fields(vocab: antu.io.vocabulary.Vocabulary, dynamic_fields: Set[str]) →Dict[str, Dict[str, Indices]]
Indexes all fields in this Instance using the provided Vocabulary. This mutates the current object, it
does not return a new Instance. A DataIterator will call this on each pass through a dataset; weuse the indexed flag to make sure that indexing only happens once. This means that if for some reason
you modify your vocabulary after you’ve indexed your instances, you might get unexpected behavior.
Parameters
vocab [Vocabulary] vocab is used to get the index of each item.
Returns
res [Dict[str, Dict[str, Indices]]] Returns the Indices corresponding to theinstance. The first key is field name and the second key is the vocabulary name.
index_fields(vocab: antu.io.vocabulary.Vocabulary) → Dict[str, Dict[str, Indices]]
Indexes all fields in this Instance using the provided Vocabulary. This mutates the current object, it
does not return a new Instance. A DataIterator will call this on each pass through a dataset; weuse the indexed flag to make sure that indexing only happens once. This means that if for some reason
you modify your vocabulary after you’ve indexed your instances, you might get unexpected behavior.
Parameters
vocab [Vocabulary] vocab is used to get the index of each item.
Returns
res [Dict[str, Dict[str, Indices]]] Returns the Indices corresponding to theinstance. The first key is field name and the second key is the vocabulary name.

10

Chapter 1. antu.io package

antu Documentation, Release 0.0.1

1.4 antu.io.vocabulary moduleclass antu.io.vocabulary.Vocabulary(counters: Dict[str, Dict[str, int]] = {}, min_count:Dict[str, int] = {}, pretrained_vocab:
Dict[str,
List[str]] = {}, intersection_vocab: Dict[str, str] = {},no_pad_namespace: Set[str] = {}, no_unk_namespace:Set[str] = {})Bases: object
Parameterscounters [Dict[str, Dict[str, int]], optional (default= dict() )] Element statistics for datasets.
min_count [Dict[str, int], optional (default= dict() )] Defines the minimum numberof occurrences when some counter are converted to vocabulary.
pretrained_vocab [Dict[str, List[str]], optional (default= dict()] External pretrained vocabulary.
intersection_vocab [Dict[str, str], optional (default= dict() )] Defines the intersection with which vocabulary takes, when loading some oversized pre-trained vocabulary.
no_pad_namespace [Set[str], optional (default= set() )] Defines which vocabularies do
not have pad token.
no_unk_namespace [Set[str], optional (default= set() )] Defines which vocabularies do
not have oov token.
Methods
add_token_to_namespace(token, namespace)extend_from_counter(counters, Dict[str, . . . )extend_from_pretrained_vocab(. . . )get_token_from_index(index, vocab_name)
get_token_index(token, vocab_name)get_vocab_size(namespace)

Extend the vocabulary by add token to vocabularynamespace.
Extend the vocabulary from the dataset statistic
counters after defining the vocabulary.
Extend the vocabulary from the pre-trained vocabulary after defining the vocabulary.
Gets the token of a index in the vocabulary.
Gets the index of a token in the vocabulary.
Gets the size of a vocabulary.

get_padding_index
get_unknow_index
add_token_to_namespace(token: str, namespace: str) → None
Extend the vocabulary by add token to vocabulary namespace.
Parameters
token [str] The token that needs to be added.
namespace [str] Which vocabulary needs to be added to.
extend_from_counter(counters: Dict[str, Dict[str, int]], min_count: Union[int, Dict[str, int]] = {},no_pad_namespace: Set[str] = {}, no_unk_namespace: Set[str] = {}) →None

1.4. antu.io.vocabulary module11antu Documentation, Release 0.0.1

Extend the vocabulary from the dataset statistic counters after defining the vocabulary.
Parameterscounters [Dict[str, Dict[str, int]]] Element statistics for datasets.
min_count [Dict[str, int], optional (default= dict() )] Defines the minimum
number of occurrences when some counter are converted to vocabulary.
no_pad_namespace [Set[str], optional (default= set() )] Defines which vocabularies
do not have pad token.
no_unk_namespace [Set[str], optional (default= set() )] Defines which vocabularies
do not have oov token.
extend_from_pretrained_vocab(pretrained_vocab: Dict[str, List[str]], intersection_vocab:
Dict[str, str] = {}, no_pad_namespace: Set[str] = {},no_unk_namespace: Set[str] = {}) → None
Extend the vocabulary from the pre-trained vocabulary after defining the vocabulary.
Parameters
pretrained_vocab [Dict[str, List[str]]] External pre-trained vocabulary.
intersection_vocab [Dict[str, str], optional (default= dict() )] Defines the intersection with which vocabulary takes, when loading some oversized pre-trained vocabulary.
no_pad_namespace [Set[str], optional (default= set() )] Defines which vocabularies
do not have pad token.
no_unk_namespace [Set[str], optional (default= set() )] Defines which vocabularies
do not have oov token.
get_padding_index(namespace: str) → int
get_token_from_index(index: int, vocab_name: str) → str
Gets the token of a index in the vocabulary.
Parametersindex [int] Gets the token of which index.
namespace [str] Which vocabulary this index belongs to.
Returns
Token [str]get_token_index(token: str, vocab_name: str) → int
Gets the index of a token in the vocabulary.
Parameters
token [str] Gets the index of which token.
namespace [str] Which vocabulary this token belongs to.
Returns
Index [int]get_unknow_index(namespace: str) → int
get_vocab_size(namespace: str) → int
Gets the size of a vocabulary.
Parameters

12

Chapter 1. antu.io package

antu Documentation, Release 0.0.1

namespace [str] Which vocabulary.
Returns
Vocabulary size [int]1.5 Module contents

1.5. Module contents

13

antu Documentation, Release 0.0.1

14Chapter 1. antu.io package

CHAPTER

2

antu.nn package

2.1 Subpackages
2.1.1 antu.nn.dynet package
Submodules
antu.nn.dynet.attention_mechanism module
antu.nn.dynet.char2word_embedder module
antu.nn.dynet.initializer module
antu.nn.dynet.multi_layer_perception module
antu.nn.dynet.nn_classifier module
antu.nn.dynet.rnn_builder module
Module contents

2.2 Module contents

15

antu Documentation, Release 0.0.1

16Chapter 2. antu.nn package

CHAPTER

3

Indices and tables• genindex• modindex• search

17

antu Documentation, Release 0.0.1

18

Chapter 3. Indices and tables

Python Module Indexaantu.io, 13
antu.io.dataset_readers, 3
antu.io.dataset_readers.dataset_reader,
3
antu.io.datasets, 4antu.io.datasets.dataset, 4antu.io.fields, 7
antu.io.fields.field, 4
antu.io.fields.index_field, 5
antu.io.fields.sequence_label_field, 5
antu.io.fields.text_field, 6
antu.io.instance, 9antu.io.token_indexers, 9
antu.io.token_indexers.char_token_indexer,
7
antu.io.token_indexers.single_id_token_indexer,
8
antu.io.token_indexers.token_indexer, 9antu.io.vocabulary, 11antu.nn, 15

19antu Documentation, Release 0.0.1

20

Python Module Index

IndexA

5

count_vocab_items()
add_field() (antu.io.instance.Instance method), 10(antu.io.fields.sequence_label_field.SequenceLabelField
add_token_to_namespace()
method), 5(antu.io.vocabulary.Vocabulary
method),count_vocab_items()
11(antu.io.fields.text_field.TextField
method),antu.io (module), 13
6antu.io.dataset_readers (module), 3
antu.io.dataset_readers.dataset_reader count_vocab_items() (antu.io.instance.Instancemethod), 10(module), 3
count_vocab_items()
antu.io.datasets (module), 4(antu.io.token_indexers.char_token_indexer.CharTokenIndexer
antu.io.datasets.dataset (module), 4
method), 7
antu.io.fields (module), 7
count_vocab_items()
antu.io.fields.field (module), 4(antu.io.token_indexers.single_id_token_indexer.SingleIdTokenInd
antu.io.fields.index_field (module), 5
method), 8
antu.io.fields.sequence_label_fieldcount_vocab_items()(module), 5(antu.io.token_indexers.token_indexer.TokenIndexer
antu.io.fields.text_field (module), 6
method), 9
antu.io.instance (module), 9
antu.io.token_indexers (module), 9D
antu.io.token_indexers.char_token_indexer
Dataset (class in antu.io.datasets.dataset), 4(module), 7
DatasetReader(classinantu.io.token_indexers.single_id_token_indexer
antu.io.dataset_readers.dataset_reader),(module), 8
3antu.io.token_indexers.token_indexer
datasets (antu.io.datasets.dataset.Dataset attribute),(module), 9
4antu.io.vocabulary (module), 11
dynamic_index_fields()
antu.nn (module), 15(antu.io.instance.Instance method), 10

B

build_dataset() (antu.io.datasets.dataset.Dataset E
extend_from_counter()method), 4(antu.io.vocabulary.Vocabulary
11
Cextend_from_pretrained_vocab()
CharTokenIndexer(classin(antu.io.vocabulary.Vocabularyantu.io.token_indexers.char_token_indexer), 7
12count_vocab_items()(antu.io.fields.field.Field

method),method),method), 4
Fcount_vocab_items()(antu.io.fields.index_field.IndexField method), Field (class in antu.io.fields.field), 4

21

antu Documentation, Release 0.0.1

G

tokens_to_indices()(antu.io.token_indexers.token_indexer.TokenIndexer
method), 9
method),get_padding_index()(antu.io.vocabulary.Vocabulary
12V
get_token_from_index()(antu.io.vocabulary.Vocabulary
method), Vocabulary (class in antu.io.vocabulary), 11
vocabulary_set (antu.io.datasets.dataset.Dataset
12attribute), 4
get_token_index()(antu.io.vocabulary.Vocabulary
method),12
get_unknow_index()(antu.io.vocabulary.Vocabulary
method),12
get_vocab_size() (antu.io.vocabulary.Vocabulary
method), 12

I
index() (antu.io.fields.field.Field method), 4
index() (antu.io.fields.index_field.IndexField method),5index() (antu.io.fields.sequence_label_field.SequenceLabelField
method), 6
index() (antu.io.fields.text_field.TextField method), 6
index_fields() (antu.io.instance.Instance method),10
IndexField (class in antu.io.fields.index_field), 5
input_to_instance()(antu.io.dataset_readers.dataset_reader.DatasetReader
method), 3
Instance (class in antu.io.instance), 9R
read() (antu.io.dataset_readers.dataset_reader.DatasetReader
method), 3

S
SequenceLabelField(classinantu.io.fields.sequence_label_field), 5
SingleIdTokenIndexer(classinantu.io.token_indexers.single_id_token_indexer),
8

T
TextField (class in antu.io.fields.text_field), 6
TokenIndexer(classinantu.io.token_indexers.token_indexer), 9
tokens_to_indices()(antu.io.token_indexers.char_token_indexer.CharTokenIndexer
method), 7
tokens_to_indices()(antu.io.token_indexers.single_id_token_indexer.SingleIdTokenIndexer
method), 8

22

Index