متن زیر رو به فارسی ترجمه کن.
The text corpus is from *all* the www, taken several years ago, plus more
recent data from wiki-something.
The text is divided by language automatically, so there is a separate
stream for each of the Devanagari-based languages (as there is for the
Latin-based languages) and clipped to 1GB for each language.
For each language, the text is frequency counted and cleaned by multiple
methods, and sometimes this cleaning is too stringent automatically, or not
stringent enough, so forbidden_characters and desired_characters are used
as a guide in the cleanup process. There are other lang-specific numbers
like a 1-in-n discard ratio for the frequency.
For some languages, the amount of data produced at the end is very thin.

The unicharset is extracted from what remains, and the wordlist that is
published in langdata.
For the LSTM training, I resorted to using Google's parallel infrastructure
to render enough text in all the languages.
However much or little corpus text there is, the rendering process makes
50000 chunks of 50 words to render in a different combination of font and
random degradation, which results in 400000-800000 rendered textlines.
The words are chosen to approximately echo the real frequency of conjunct
clusters (characters in most languages) in the source text, while also
using the most frequent words.

This process is all done without significant manual intervention, but
counts of the number of generated textlines indicates when it has gone
badly, usually due to a lack of fonts, or a lack of corpus text.
I recently stopped training chr, iku, khm, mya after discovering that I
have no rendered textlines that contain anything other than digits and
punctuation.

Community input is therefore extremely useful, and usually results in edits
to forbidden_characters and desired_characters, which in turn guides the
filtration process.
Community-provided corpus text would be useful for languages that have very
little or no training data, given appropriate copyright/licensing clearance.
The languages with very little corpus text are:
bih
chr
dzo
iku
snd
syr
tgk
tir
so these are likely to have poor recognition accuracy.

On Sat, Jan 21, 2017 at 7:46 AM, Shreeshrii <<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>>
wrote:
…

 Ray,

 Thank you for explaining regrading unicharset compression and your new
 strategy for Indic graphemes.

 Since the unicharset is being used as a filter, it will be important to
 include the most common conjunct clusters in it, which may differ from
 language to language.

 Some more questions

 Are the desired_characters and forbidden_characters used in the process of
 creating the text corpus for different languages?

 How many text lines are you using for training of Devanagari, e.g.
 Sanskrit, Hindi, Marathi etc. Is it all/only from Wikipedia?



 - excuse the brevity, sent from mobile

 On 21-Jan-2017 3:34 AM, "theraysmith" ***@***.***> wrote:

 > The LSTM recognizer is currently trained to recognize the sequence of
 > *unicodes* for Indic languages. This reduces the size of the output
 > softmax of the network from the 5000+ elements in the unicharset to ~140.
 > (There is an analogous process for Chinese, Japanese, and Korean, that
 > doesn't use the unicode encoding, but it is a similar idea, and the codes
 > are strictly limited in length.)
 > The unicharset is used as a *filter* in the beam search to allow only
 > sensible grapheme/syllable combinations of unicodes, so it doesn't output
 > complete garbage text.
 >
 > The consequence of this recoding is that it runs a lot faster, but it has
 > to learn to output a long sequence for each grapheme/syllable.
 > The recoding system that maps from unicharset elements to the sequence of
 > unicodes currently only allows a maximum of 9 unicodes per
 > grapheme/syllable, including any viramas.
 >
 > I'm running a new training experiment this weekend to try a new coding
 > scheme, in which pairs are mapped to a single code, allowing a long
 CVCVCVC
 > string to be encoded using just CCCC, cutting down from 7 codes to 4.
 This
 > will probably increase the size of the output softmax to ~170, but reduce
 > the length of the average code sequence by about 1/3, which might be
 easier
 > for it to learn, without slowing it down much.
 >
 > It will take a couple of weeks to tell if it works, but if it does I will
 > check in the code, and upload new traineddatas, and close this issue. If
 it
 > doesn't work, I will have to think again...
 >
 > —
 > You are receiving this because you authored the thread.
 > Reply to this email directly, view it on GitHub
 > <#654#
 issuecomment-274192153>,
 > or mute the thread
 > <https://github.com/notifications/unsubscribe-auth/AE2_o-xusyCIFbh-
 wE4T4cp4mVb4oBWWks5rUS9vgaJpZM4LhbNY>
 > .

 >

 —
 You are receiving this because you were mentioned.
 Reply to this email directly, view it on GitHub
 <#654 (comment)>,
 or mute the thread
 <https://github.com/notifications/unsubscribe-auth/AL056XOUmyQKlAM4aHUJc-jTRmhEwWOxks5rUihVgaJpZM4LhbNY>
 .


-- 
Ray.