Package: nlptext
.
├── __init__.py
├── base.py
├── corpus.py
├── folder.py
├── text.py
├── sentence.py
├── token.py
└── utils
    ├── channel.py
    ├── grain.py
    ├── infrastructure.py
    └── pyramid.py1. PyramidFrom Corpus to Groups to Texts to Sentences to TokensSaveCOPRUS,GROUP,TEXT,SENTandTOKENSave token tofile/token.txt.Get Token Vocabulary (idx2token,token2index), and token frequency (idx2freq).2. QueryConstruct a connection across corpus, group, text, sentence and token.Given a sentence Idx (the location index), get the sentence contentGiven a text Idx (the location index), get the text content.Given a token Idx (the location index), get the token content.3. AnnotationBy whatever means, from group to texts, for each texts, we need to prepare[strText:str, SSET:[content:str, start:int, end:int, label: str]]. Besides, we need to verify that SSET is valid for strText.Then strText is processed tostrSents: [strSent:str].Together with strText and strSents, we use SSET to get the tag sequence for each strSent. The tag is in the BIOES tag scheme. However, instead of saving the tag string, we save the tag idx this time.Save the token tofile/annoE_idx.txt. Each line is the tag seqeuence correponding to the line of the original sentence infile/token.txt.We also need to query the annotation tag sequence from that file.Build theBIOES_trans. This is a dictioanry. The formal content is bioesidx2bioidx. Its path should beBIOES_trans['annoE']['BIO'].4. Hyper Field of Token Granularity (i.e. part-of-speech, word level)Hyper field of token granularity means the hyper field entity corresponds to exactly one token entity.For example, a part-of-speech entity corresponds to exactly one word. When the token level is word, part-of-speech is the hyper field of token granularity.We need to develop a function to get hyper field grains for a given sentence.This is on the process of sentence to tokens. Input is strSent: str, and output istokens: [token:str]andtags: [tag:str], where tag is of BIOES tagging scheme. Similar to annotation, we also only save the tag idx and save all of them into file/pos_idx.txt.We also need to query the annotation tag sequence from that file.Build theBIOES_trans. This is a dictioanry. The formal content is bioesidx2bioidx. Its path should beBIOES_trans['pos']['BIO'].5. Hyper Field of Larger Granularity (i.e. named entitiy)Hyper field whose granularity larger than token means the hyper field entity corresponds to more than (including equal to) one token entity.We need to develop a function to get hyper field grains for a given sentence.This is on the process of sentence to tokens. Input is strSent: str, and output is tags[tag:str], where tag is of BIOES tagging scheme. Similar to annotation, we also only save the tag idx and save all of them into file/pos_idx.txt.We also need to query the annotation tag sequence from that file.Build theBIOES_trans. This is a dictioanry. The formal content is bioesidx2bioidx. Its path should beBIOES_trans['ner']['BIO'].6. Sub Field InformationSub field information refers to the grains that only determined exclusively by a token, such as the letter sequence of an English word.We need to develop a function to get sub field grains for a given sentence.Get Sub field Grain Vocabulary based on the token vocabulary, and the n-gram we choose.Get the frequency tables of the sub field grains based on the whole corpus and on the whole token vocabulary.Get the Look Up Table to connect token vocabulary and sub field vocabulary.7. Query Field InformationFor sentences and tokens, whether they are from our corpus pyramid or are newly created, given a field, get their field grain sequence.get_grain_strget_grain_idxThey are APIs for further tasks.8. Chunk of Texts or SentencesSometimes we need a chunk of texts or sentences where the number of tokens numbers in this chunk is close to and smaller than a constant, say, 10000. One scenario is for word embedding training.Use text as an instance, the data we want is: chunkidx2endtxtidx.For each chunk, get all text idxs in it. From startidx to endidx.The following still uses text as the instance. The concrete operation:chunkidx_2_endbyteidx = [] # the text that is not included inside this chunk.
chunkidx_2_cumlengoftexts = []

current_chunk_lengoftexts = []
current_chunk_length_count = 0
for text in texts:
	text_token_num = text.get_tokens_num()

	if current_chunk_length_count + text_token_num > 10000:

		# get and save the chunkendbyteidx
		# this is correct, notice that this text is not include in this chunk

		endbyteidx = text.get_endbyteidx() 
		chunkidx_2_endbyteidx.append(endbyteidx)

		# save the txtidx2endtokenidx, derived from lengoftexts
		chunkidx_2_cumlengoftexts.append(np.cumsum(current_chunk_lengoftexts))

		# get the new the current chunk information
		current_chunk_length_count = text_token_num
		current_chunk_lengoftexts = [text_token_num]

	else:
		# update current_chunk_length_count when it is <= 10000
		current_chunk_length_count = current_chunk_length_count + text_token_num

		# append the object's length in the current_chunk_lengoftexts
		current_chunk_lengoftexts.append(text_token_num)

# when the loop is over, we still need the to append 
# the last small chunk into the total chunks.
# here, the object is the smallest object.
endbyteidx = object.get_endbyteidx()
chunkidx_2_endbyteidx.append(endbyteidx)

# save the txtidx2endtokenidx, derived from lengoftexts
chunkidx_2_cumlengoftexts.append(np.cumsum(current_chunk_lengoftexts))Eventually, we get chunkidx_2_endbyteidx and chunkidx_2_cumlengoftexts.Then we should think out how to produce jobs. Each chunk is for one job.# the file we need to read is of path: path_token
job_paras = []
for chunkidx in range(len(chunkidx_2_endbyteidx)):
	startbyteidx = 0 if chunkidx == 0 else chunkidx_2_endbyteidx[chunkidx-1]
	endbyteidx = chunkidx_2_endbyteidx[chunkidx-1]
	chunk_token_string = read_str_from_file(path_token, startbyteidx, endbyteidx)
	# for the text in the chunk.
	txtidx2endtokenidx = chunkidx_2_lengoftexts[chunk_string]
	job_para = (chunk_token_string, txtidx2endtokenidx)
	job_paras.append(job_para)Inside the fieldembed_core.pyx, the method is:# for the input: chunk_tokens (i.e. chunk_token_string), txtidx2endtokenidx.
# we first prepare chunk_token_seq.

chunk_tokens = chunk_tokens.split() # i.e. chunk_token_seq

# then loop the text sequence in chunk_tokens

token_indexes = []
text_endidx   = []

new_chunk_idx = 0


for txtidx, endtokenidx in enumerate(txtidx2endtokenidx):
	starttokenidx = 0 if txtidx == 0 else txtidx2endtokenidx[endtokenidx - 1]
	for tokenidx in range(starttokenidx, endtokenidx):
		tokenstr = get_token_str(tokenidx)
		tokenvocidx = get_token_vocidx(tokenstr)

		if tokenvocidx not meets conditions:
			continue

		token_indexes.append(tokenvocidx)
		new_chunk_idx = new_chunk_idx + 1

	text_endidx.append(new_chunk_idx)

# at the end, we get token_indexes, text_endidx, new_chunk_idx.