Upload 39 files

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+marian-server-GPU filter=lfs diff=lfs merge=lfs -text
+start filter=lfs diff=lfs merge=lfs -text
+stop filter=lfs diff=lfs merge=lfs -text
+tc.es filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,674 @@
+                    GNU GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The GNU General Public License is a free, copyleft license for
+software and other kinds of works.
+
+  The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works.  By contrast,
+the GNU General Public License is intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.  We, the Free Software Foundation, use the
+GNU General Public License for most of our software; it applies also to
+any other work released this way by its authors.  You can apply it to
+your programs, too.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+  To protect your rights, we need to prevent others from denying you
+these rights or asking you to surrender the rights.  Therefore, you have
+certain responsibilities if you distribute copies of the software, or if
+you modify it: responsibilities to respect the freedom of others.
+
+  For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must pass on to the recipients the same
+freedoms that you received.  You must make sure that they, too, receive
+or can get the source code.  And you must show them these terms so they
+know their rights.
+
+  Developers that use the GNU GPL protect your rights with two steps:
+(1) assert copyright on the software, and (2) offer you this License
+giving you legal permission to copy, distribute and/or modify it.
+
+  For the developers' and authors' protection, the GPL clearly explains
+that there is no warranty for this free software.  For both users' and
+authors' sake, the GPL requires that modified versions be marked as
+changed, so that their problems will not be attributed erroneously to
+authors of previous versions.
+
+  Some devices are designed to deny users access to install or run
+modified versions of the software inside them, although the manufacturer
+can do so.  This is fundamentally incompatible with the aim of
+protecting users' freedom to change the software.  The systematic
+pattern of such abuse occurs in the area of products for individuals to
+use, which is precisely where it is most unacceptable.  Therefore, we
+have designed this version of the GPL to prohibit the practice for those
+products.  If such problems arise substantially in other domains, we
+stand ready to extend this provision to those domains in future versions
+of the GPL, as needed to protect the freedom of users.
+
+  Finally, every program is threatened constantly by software patents.
+States should not allow patents to restrict development and use of
+software on general-purpose computers, but in those that do, we wish to
+avoid the special danger that patents applied to a free program could
+make it effectively proprietary.  To prevent this, the GPL assures that
+patents cannot be used to render the program non-free.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+                       TERMS AND CONDITIONS
+
+  0. Definitions.
+
+  "This License" refers to version 3 of the GNU General Public License.
+
+  "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+  "The Program" refers to any copyrightable work licensed under this
+License.  Each licensee is addressed as "you".  "Licensees" and
+"recipients" may be individuals or organizations.
+
+  To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy.  The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+  A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+  To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy.  Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+  To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies.  Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+  An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License.  If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+  1. Source Code.
+
+  The "source code" for a work means the preferred form of the work
+for making modifications to it.  "Object code" means any non-source
+form of a work.
+
+  A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+  The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form.  A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+  The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities.  However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work.  For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+  The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+  The Corresponding Source for a work in source code form is that
+same work.
+
+  2. Basic Permissions.
+
+  All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met.  This License explicitly affirms your unlimited
+permission to run the unmodified Program.  The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work.  This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+  You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force.  You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright.  Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+  Conveying under any other circumstances is permitted solely under
+the conditions stated below.  Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+  No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+  When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+  4. Conveying Verbatim Copies.
+
+  You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+  You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+  5. Conveying Modified Source Versions.
+
+  You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+    a) The work must carry prominent notices stating that you modified
+    it, and giving a relevant date.
+
+    b) The work must carry prominent notices stating that it is
+    released under this License and any conditions added under section
+    7.  This requirement modifies the requirement in section 4 to
+    "keep intact all notices".
+
+    c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy.  This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged.  This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+
+    d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+  A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit.  Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+  6. Conveying Non-Source Forms.
+
+  You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+    a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+
+    b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the
+    Corresponding Source from a network server at no charge.
+
+    c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source.  This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in accord
+    with subsection 6b.
+
+    d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge.  You need not require recipients to copy the
+    Corresponding Source along with the object code.  If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source.  Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+
+    e) Convey the object code using peer-to-peer transmission, provided
+    you inform other peers where the object code and Corresponding
+    Source of the work are being offered to the general public at no
+    charge under subsection 6d.
+
+  A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+  A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling.  In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage.  For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product.  A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+  "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source.  The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+  If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information.  But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+  The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed.  Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+  Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+  7. Additional Terms.
+
+  "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law.  If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+  When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it.  (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.)  You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+  Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+    a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+
+    b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+
+    c) Prohibiting misrepresentation of the origin of that material, or
+    requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+
+    d) Limiting the use for publicity purposes of names of licensors or
+    authors of the material; or
+
+    e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+
+    f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions of
+    it) with contractual assumptions of liability to the recipient, for
+    any liability that these contractual assumptions directly impose on
+    those licensors and authors.
+
+  All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10.  If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term.  If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+  If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+  Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+  8. Termination.
+
+  You may not propagate or modify a covered work except as expressly
+provided under this License.  Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+  However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+  Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+  Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License.  If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+  9. Acceptance Not Required for Having Copies.
+
+  You are not required to accept this License in order to receive or
+run a copy of the Program.  Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance.  However,
+nothing other than this License grants you permission to propagate or
+modify any covered work.  These actions infringe copyright if you do
+not accept this License.  Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+  10. Automatic Licensing of Downstream Recipients.
+
+  Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License.  You are not responsible
+for enforcing compliance by third parties with this License.
+
+  An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations.  If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+  You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License.  For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+  11. Patents.
+
+  A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based.  The
+work thus licensed is called the contributor's "contributor version".
+
+  A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version.  For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+  Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+  In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement).  To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+  If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients.  "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+  If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+  A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License.  You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+  Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+  12. No Surrender of Others' Freedom.
+
+  If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all.  For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+  13. Use with the GNU Affero General Public License.
+
+  Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU Affero General Public License into a single
+combined work, and to convey the resulting work.  The terms of this
+License will continue to apply to the part which is the covered work,
+but the special requirements of the GNU Affero General Public License,
+section 13, concerning interaction through a network will apply to the
+combination as such.
+
+  14. Revised Versions of this License.
+
+  The Free Software Foundation may publish revised and/or new versions of
+the GNU General Public License from time to time.  Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+  Each version is given a distinguishing version number.  If the
+Program specifies that a certain numbered version of the GNU General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation.  If the Program does not specify a version number of the
+GNU General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+  If the Program specifies that a proxy can decide which future
+versions of the GNU General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+  Later license versions may give you additional or different
+permissions.  However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+  15. Disclaimer of Warranty.
+
+  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+  16. Limitation of Liability.
+
+  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+  17. Interpretation of Sections 15 and 16.
+
+  If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+  If the program does terminal interaction, make it output a short
+notice like this when it starts in an interactive mode:
+
+    <program>  Copyright (C) <year>  <name of author>
+    This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+    This is free software, and you are welcome to redistribute it
+    under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License.  Of course, your program's commands
+might be different; for a GUI interface, you would use an "about box".
+
+  You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU GPL, see
+<https://www.gnu.org/licenses/>.
+
+  The GNU General Public License does not permit incorporating your program
+into proprietary programs.  If your program is a subroutine library, you
+may consider it more useful to permit linking proprietary applications with
+the library.  If this is what you want to do, use the GNU Lesser General
+Public License instead of this License.  But first, please read
+<https://www.gnu.org/licenses/why-not-lgpl.html>.

MTUOC-server.py

@@ -0,0 +1,308 @@
+#    MTUOC-server v 6
+#    Description: an MTUOC server using Sentence Piece as preprocessing step
+#    Copyright (C) 2023  Antoni Oliver
+#    v. 11/09/2023
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+import os
+import sys
+import asyncio
+import websockets
+import argparse
+import codecs
+import csv
+
+import srx_segmenter
+
+
+from websocket import create_connection
+import socket
+import time
+import asyncio
+import websockets
+
+import config
+
+from MTUOC_misc import printLOG
+from MTUOC_misc import get_IP_info
+
+from MTUOC_Marian import connect_to_Marian
+from MTUOC_Marian import translate_segment_Marian
+
+from MTUOC_OpenNMT import connect_to_OpenNMT
+from MTUOC_OpenNMT import translate_segment_OpenNMT
+
+from MTUOC_Moses import connect_to_Moses
+from MTUOC_Moses import translate_segment_Moses
+
+from MTUOC_typeMTUOC import start_MTUOC_server
+from MTUOC_typeMoses import start_Moses_server
+from MTUOC_typeOpenNMT import start_OpenNMT_server
+from MTUOC_typeNMTWizard import start_NMTWizard_server
+from MTUOC_typeModernMT import start_ModernMT_server
+
+
+
+
+from MTUOC_tags import TagRestorer
+
+
+###YAML IMPORTS
+import yaml
+from yaml import load, dump
+try:
+    from yaml import CLoader as Loader, CDumper as Dumper
+except ImportError:
+    from yaml import Loader, Dumper
+
+def startMTEngine():
+    printLOG(3,"START MT ENGINE:", config.startMTEngineCommand)
+    os.system(config.startMTEngineCommand)   
+
+
+ #YAML
+
+parser = argparse.ArgumentParser(description='MTUOC-server. With no arguments the config-server.yaml file will be used.')
+parser.add_argument('-c','--config', action="store", dest="config", help='The server configuration file to be used.',required=False)
+parser.add_argument('-p','--port', action="store", dest="port", type=int, help='The MTUOC server port.',required=False)
+parser.add_argument('-t','--type', action="store", dest="type", help='The MTUOC server type.',required=False)
+
+
+args = parser.parse_args()
+if args.config:
+    configfile=args.config
+else:
+    configfile="config-server.yaml"
+
+stream = open(configfile, 'r',encoding="utf-8")
+configYAML=yaml.load(stream, Loader=yaml.FullLoader)
+
+config.MTUOCServer_MTengine=configYAML["MTEngine"]["MTengine"]
+config.MTUOCServer_type=configYAML["MTUOCServer"]["type"]
+config.MTUOCServer_port=configYAML["MTUOCServer"]["port"]
+config.tag_restoration=configYAML["MTUOCServer"]["restore_tags"]
+config.fix_xml=configYAML["MTUOCServer"]["fix_xml"]
+
+config.MTUOCServer_URLs=configYAML["MTUOCServer"]["URLs"]
+config.MTUOCServer_EMAILs=configYAML["MTUOCServer"]["EMAILs"]
+
+config.code_URLs=configYAML["MTUOCServer"]["code_URLs"]
+config.code_EMAILs=configYAML["MTUOCServer"]["code_EMAILs"]
+
+config.min_chars_segment=configYAML["MTUOCServer"]["min_chars_segment"]
+translation_selection_strategy=configYAML["MTUOCServer"]["translation_selection_strategy"]
+
+config.segment_input=configYAML["Preprocess"]["segment_input"]
+config.SRXfile=configYAML["Preprocess"]["SRXfile"]
+config.SRXlang=configYAML["Preprocess"]["SRXlang"]
+
+config.rules = srx_segmenter.parse(config.SRXfile)
+
+config.sllang=configYAML["Preprocess"]["sl_lang"]
+config.tllang=configYAML["Preprocess"]["tl_lang"]
+config.tokenize_SL=configYAML["Preprocess"]["tokenize_SL"]
+config.tokenizerSL=configYAML["Preprocess"]["sl_tokenizer"]
+tokenizerSLfile=config.tokenizerSL
+config.tokenizerTL=configYAML["Preprocess"]["tl_tokenizer"]
+if config.tokenizerSL=="None": config.tokenizerSL=None
+if config.tokenizerTL=="None": config.tokenizerTL=None
+
+if config.tokenizerSL=="Moses":
+    config.tokenizerSLType="Moses"
+    import mosestokenizer
+    config.tokenizerSL = mosestokenizer.MosesTokenizer(config.sllang)
+    config.tokenizerSLType="Moses"
+elif not config.tokenizerSL==None:
+    config.tokenizerSLType="MTUOC"
+    import importlib.util
+    if not config.tokenizerSL.endswith(".py"): config.tokenizerSL=MTUOCtokenizerSL+".py"
+    spec = importlib.util.spec_from_file_location('', config.tokenizerSL)
+    tokenizerSLmod = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(tokenizerSLmod)
+    config.tokenizerSL=tokenizerSLmod.Tokenizer()
+    
+else:
+    config.tokenizerSL=None
+    
+if config.tokenizerTL=="Moses":
+    import mosestokenizer
+    config.tokenizerTL = mosestokenizer.MosesTokenizer(config.tllang)
+    config.tokenizerTLType="Moses"
+elif not config.tokenizerTL==None:
+    import importlib.util
+    if not config.tokenizerTL.endswith(".py"): config.tokenizerTL=tokenizerTL+".py"
+    spec = importlib.util.spec_from_file_location('', config.tokenizerTL)
+    tokenizerTLmod = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(tokenizerTLmod)
+    config.tokenizerTL=tokenizerTLmod.Tokenizer()
+else:
+    config.tokenizerTL=None
+    
+    
+
+config.tcmodel=configYAML["Preprocess"]["tcmodel"]
+config.truecase=configYAML["Preprocess"]["truecase"]
+
+if config.tcmodel=="None": config.tcmodel=None
+if config.tcmodel:
+    from MTUOC_truecaser import Truecaser
+    config.truecaser=Truecaser(tokenizer=tokenizerSLfile,tc_model=config.tcmodel)
+else:
+    config.truecaser=None
+
+config.unescape_html_input=configYAML["unescape_html_input"]
+config.escape_html_input=configYAML["escape_html_input"]
+
+config.checkistranslatable=configYAML["MTUOCServer"]["checkistranslatable"]
+
+config.change_input_files=configYAML["change_input_files"].split(";")
+config.changes_input=[]
+if not config.change_input_files[0]=="None":
+    for ci in config.change_input_files:
+        with open(ci) as csvfile:
+            csvreader = csv.reader(csvfile, delimiter=';', quotechar='"')
+            for row in csvreader:
+                config.changes_input.append(row)
+
+config.change_output_files=configYAML["change_output_files"].split(";")
+config.changes_output=[]
+if not config.change_output_files[0]=="None":
+    for co in config.change_output_files:
+        with open(co) as csvfile:
+            csvreader = csv.reader(csvfile, delimiter=';', quotechar='"')
+            for row in csvreader:
+                config.changes_output.append(row)
+
+config.change_translation_files=configYAML["change_translation_files"].split(";")
+config.changes_translation=[]
+if not config.change_translation_files[0]=="None":
+    for change_list in config.change_translation_files:
+        with open(change_list) as csvfile:
+            csvreader = csv.reader(csvfile, delimiter=';', quotechar='"')
+            for row in csvreader:
+                config.changes_translation.append(row)
+
+
+config.tagrestorer=TagRestorer()
+
+
+
+#sentencepiece
+config.sentencepiece=configYAML["Preprocess"]["sentencepiece"]
+config.spmodelSL=configYAML["Preprocess"]["sp_model_SL"]
+config.spmodelTL=configYAML["Preprocess"]["sp_model_TL"]
+config.sp_splitter=configYAML["Preprocess"]["sp_splitter"]
+
+#BPE
+
+#BPE
+
+config.BPE=configYAML["Preprocess"]["BPE"]
+config.bpecodes=configYAML["Preprocess"]["bpecodes"]
+config.bpe_joiner=configYAML["Preprocess"]["bpe_joiner"]
+if config.BPE:
+    from subword_nmt import apply_bpe
+    config.bpeobject=apply_bpe.BPE(open(config.bpecodes,encoding="utf-8"))
+#bos and eos annotate
+config.bos_annotate=configYAML["Preprocess"]["bos_annotate"]
+config.bos_symbol=configYAML["Preprocess"]["bos_symbol"]
+config.eos_annotate=configYAML["Preprocess"]["eos_annotate"]
+config.eos_symbol=configYAML["Preprocess"]["eos_symbol"]
+
+config.pre_replace_NUMs=configYAML["Preprocess"]["replaceNUMs"]
+config.code_NUMs=configYAML["Preprocess"]["code_NUMs"]
+config.pre_split_NUMs=configYAML["Preprocess"]["splitNUMs"]
+
+if config.sentencepiece:
+    import sentencepiece as spm
+    config.spSL= spm.SentencePieceProcessor(model_file=config.spmodelSL, out_type=str, add_bos=config.bos_annotate, add_eos=config.eos_annotate)
+    config.spTL= spm.SentencePieceProcessor(model_file=config.spmodelTL, out_type=str)
+
+
+if config.MTUOCServer_MTengine in ["Marian","Moses","OpenNMT"]:
+    config.startMTEngineV=configYAML["MTEngine"]["startMTEngine"]
+    config.startMTEngineCommand=configYAML["MTEngine"]["startCommand"]
+    config.MTEngineIP=configYAML["MTEngine"]["IP"]
+    config.MTEnginePort=configYAML["MTEngine"]["port"]
+    config.min_len_factor=configYAML["MTEngine"]["min_len_factor"]
+
+if config.MTUOCServer_MTengine=="GoogleTranslate":    
+    config.Google_sllang=configYAML["GoogleTranslate"]["sllang"]
+    config.Google_tllang=configYAML["GoogleTranslate"]["tllang"]
+    config.Google_glossary=configYAML["GoogleTranslate"]["glossary"]
+    if config.Google_glossary=="None": config.Google_glossary=None
+    #state None if no glossary is used, otherwise the name of the glossary
+    config.Google_project_id=configYAML["GoogleTranslate"]["project_id"]
+    config.Google_location=configYAML["GoogleTranslate"]["location"]
+    config.Google_jsonfile=configYAML["GoogleTranslate"]["jsonfile"]
+    os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = config.Google_jsonfile
+    config.client = translateGoogle.TranslationServiceClient()
+    config.client = translateGoogle.TranslationServiceClient()
+    config.parent = "projects/"+config.Google_project_id+"/locations/"+config.Google_location
+    
+if config.MTUOCServer_MTengine=="DeepL":
+    config.DeepL_API_key=configYAML["DeepL"]["API_key"]
+    config.DeepL_sl_lang=configYAML["DeepL"]["sllang"]
+    config.DeepL_tl_lang=configYAML["DeepL"]["tllang"]
+      #tag_handling: html
+      #one of html, xml
+    config.DeepL_formality=configYAML["DeepL"]["formality"]
+      #one of default, less, more
+    config.DeepL_split_sentences=configYAML["DeepL"]["split_sentences"]
+      #one of off, on, nonewlines
+    config.DeepL_glossary=configYAML["DeepL"]["glossary"]
+    if config.DeepL_glossary=="None": config.DeepL_glossary=None
+    config.DeepLtranslator = deepl.Translator(config.DeepL_API_key)
+
+if config.MTUOCServer_MTengine=="Lucy":
+    ###Lucy imports
+    import ast
+    import xmltodict
+    import requests
+    config.Lucy_url=configYAML["Lucy"]["url"]
+    config.Lucy_TRANSLATION_DIRECTION=configYAML["Lucy"]["TRANSLATION_DIRECTION"]
+    config.Lucy_MARK_UNKNOWNS=configYAML["Lucy"]["MARK_UNKNOWNS"]
+    config.Lucy_MARK_ALTERNATIVES=configYAML["Lucy"]["MARK_ALTERNATIVES"]
+    config.Lucy_MARK_COMPOUNDS=configYAML["Lucy"]["MARK_COMPOUNDS"]
+    config.Lucy_CHARSET=configYAML["Lucy"]["CHARSET"]
+
+config.verbosity_level=int(configYAML["MTUOCServer"]["verbosity_level"])
+config.log_file=configYAML["MTUOCServer"]["log_file"]
+
+config.MTUOCServer_ONMT_url_root=configYAML["MTUOCServer"]["ONMT_url_root"]
+
+if config.log_file=="None":
+    config.log_file=False
+else:
+    config.sortidalog=codecs.open(config.log_file,"a",encoding="utf-8")
+    config.log_file=True
+    
+if config.startMTEngineV and not config.MTUOCServer_MTengine in ["GoogleTranslate","DeepL","Lucy"]:
+    startMTEngine()
+    
+if config.MTUOCServer_MTengine=="Marian":
+    connect_to_Marian()
+elif config.MTUOCServer_MTengine=="Moses":
+    config.proxyMoses=connect_to_Moses()
+
+if config.MTUOCServer_type=="MTUOC":
+    start_MTUOC_server()
+elif config.MTUOCServer_type=="Moses":
+    start_Moses_server()
+elif config.MTUOCServer_type=="OpenNMT":
+    start_OpenNMT_server()
+elif config.MTUOCServer_type=="NMTWizard":
+    start_NMTWizard_server()
+elif config.MTUOCServer_type=="ModernMT":
+    start_ModernMT_server()

MTUOC-stop-server.py

@@ -0,0 +1,61 @@
+#    MTUOC_stop_server
+#    Copyright (C) 2023  Antoni Oliver
+#    v. 07/06/2023
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+import sys
+import os
+import yaml
+from yaml import load, dump
+try:
+    from yaml import CLoader as Loader, CDumper as Dumper
+except ImportError:
+    from yaml import Loader, Dumper
+import argparse
+
+#YAML
+
+parser = argparse.ArgumentParser(description='MTUOC-server. With no arguments the config-server.yaml file will be used.')
+parser.add_argument('-c','--config', action="store", dest="config", help='The tokenizer to be used.',required=False)
+parser.add_argument('-p','--port', action="store", dest="port", type=int, help='The MTUOC server port.',required=False)
+
+args = parser.parse_args()
+if args.config:
+    configfile=args.config
+else:
+    configfile="config-server.yaml"
+
+stream = open(configfile, 'r',encoding="utf-8")
+config=yaml.load(stream, Loader=yaml.FullLoader)
+
+MTEnginePort=config["MTEngine"]["port"]
+MTUOCServer_port=config["MTUOCServer"]["port"]
+
+if args.port:
+    MTUOCServer_port=args.port
+
+try:
+    stopcommand2="fuser -k "+str(MTEnginePort)+"/tcp"
+    os.system(stopcommand2)
+    print("MT Engine stopped.")
+except:
+    print("Unable to stop MT Engine",sys.exc_info())
+    
+try:
+    stopcommand2="fuser -k "+str(MTUOCServer_port)+"/tcp"
+    os.system(stopcommand2)
+    print("MTUOC server stopped.")
+except:
+    print("Unable to stop MTUOC server",sys.exc_info())

MTUOC_DeepL.py

@@ -0,0 +1,49 @@
+#    MTUOC_DeepL
+#    Copyright (C) 2023  Antoni Oliver
+#    v. 07/06/2023
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+import config
+from MTUOC_misc import printLOG
+
+###DeepL imports
+import deepl
+
+def DeepL_translate(segment):
+    if config.DeepL_glossary==None:
+        cadena="Translating without glossary from "+config.DeepL_sl_lang+" to "+config.DeepL_tl_lang
+        printLOG(3,cadena)
+        cadena="Source segment: "+segment
+        printLOG(3,cadena)
+        try:
+            translation = config.DeepLtranslator.translate_text(segment, source_lang=config.DeepL_sl_lang, target_lang=config.DeepL_tl_lang,formality=config.DeepL_formality,split_sentences=config.DeepL_split_sentences)
+        except:
+            printLOG(1,"ERROR DeepL:",sys.exc_info())
+        cadena="Translation:    "+translation.text
+        printLOG(3,cadena)
+    else:
+        cadena="Translating with glossary "+config.DeepL_glossary_name,DeepL_glossary+" from "+config.DeepL_sl_lang+" to "+config.DeepL_tl_lang
+        printLOG(3,cadena)
+        cadena="Source segment: "+segment
+        printLOG(3,cadena)
+        try:
+            translation = config.DeepLtranslator.translate_text(segment, source_lang=config.DeepL_sl_lang, target_lang=config.DeepL_tl_lang, glossary=config.DeepL_glossary, formality=config.DeepL_formality, split_sentences=config.DeepL_split_sentences)
+        except:
+            printLOG(1,"ERROR DeepL:",sys.exc_info())
+        cadena="Translation:    "+translation.text
+        printLOG(3,cadena)
+
+    return(translation.text)
+

MTUOC_GoogleTranslate.py

@@ -0,0 +1,83 @@
+#    MTUOC_GoogleTranslate
+#    Copyright (C) 2023  Antoni Oliver
+#    v. 07/06/2023
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from MTUOC_misc import printLOG
+import config
+
+###GoogleTranslate imports
+from google.cloud import translate as translateGoogle
+from google.cloud import translate
+
+def Google_translate_text_with_glossary(text):
+    
+    glossary = config.client.glossary_path(
+        config.Google_project_id, config.Google_location, config.Google_glossary 
+    )
+
+    glossary_config = translateGoogle.TranslateTextGlossaryConfig(glossary=config.Google_glossary)
+
+    # Supported language codes: https://cloud.google.com/translate/docs/languages
+    response = client.translate_text(
+        request={
+            "contents": [text],
+            "target_language_code": config.Google_sllang,
+            "source_language_code": config.Google_tllang,
+            "parent": config.parent,
+            "glossary_config": glossary_config,
+        }
+    )
+    translation=response.glossary_translations[0]
+    return(translation.translated_text)
+
+def Google_translate_text(text):
+    try:
+        """Translating Text. segment,config.Google_sllang,config.Google_tllang"""
+        # Detail on supported types can be found here:
+        # https://cloud.google.com/translate/docs/supported-formats
+        response = config.client.translate_text(
+            parent=config.parent,
+            contents=[text],
+            mime_type="text/plain",  # mime types: text/plain, text/html
+            source_language_code=config.Google_sllang,
+            target_language_code=config.Google_tllang,
+        )
+        # Display the translation for each input text provided
+        translation=response.translations[0]
+        return(translation.translated_text)
+    except:
+        print("ERROR:",sys.exc_info())
+
+    
+def Google_translate(segment):
+    segment=segment.rstrip()
+    if config.Google_glossary==None:
+        cadena="Translating without glossary from "+config.Google_sllang+" to "+config.Google_tllang
+        printLOG(3,cadena)
+        cadena="Source segment: "+segment
+        printLOG(3,cadena)
+        translation=Google_translate_text(segment)
+        cadena="Translation:    "+translation
+        printLOG(3,cadena)
+    else:
+        cadena="Translating with glossary "+Google_glossary+" from "+config.Google_sllang+" to "+config.Google_tllang
+        printLOG(3,cadena)
+        cadena="Source segment: "+segment
+        printLOG(3,cadena)
+        translation=Google_translate_text_with_glossary()
+        cadena="Translation:    "+translation
+        printLOG(3,cadena)
+    return(translation)

MTUOC_Lucy.py

@@ -0,0 +1,40 @@
+#    MTUOC_Lucy
+#    Copyright (C) 2023  Antoni Oliver
+#    v. 07/06/2023
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+import config
+from MTUOC_misc import printLOG
+
+import json
+import requests
+import ast
+import xmltodict
+
+def Lucy_translate(slsegment):
+    try:
+        slsegment="[@àéèíóòú@]. "+slsegment
+        url = config.Lucy_url 
+        headers = {'Content-Type': 'application/json'}
+        body = {"inputParams": {"param": [{"@name": "TRANSLATION_DIRECTION", "@value": config.Lucy_TRANSLATION_DIRECTION},{"@name": "MARK_UNKNOWNS", "@value": config.Lucy_MARK_UNKNOWNS},{"@name": "MARK_ALTERNATIVES", "@value": config.Lucy_MARK_ALTERNATIVES},{"@name": "MARK_COMPOUNDS", "@value": config.Lucy_MARK_COMPOUNDS}, {"@name": "INPUT", "@value": slsegment},{"@name": "CHARSET","@value":config.Lucy_CHARSET}]}}
+        encoded_body = json.dumps(body)
+        traduccio_xml=requests.post(url = url, auth=('traductor', 'traductor'), headers=headers, data=encoded_body).text
+        response_dict = ast.literal_eval(json.dumps(xmltodict.parse(traduccio_xml)))
+        translation=response_dict["task"]["outputParams"]["param"][0]['@value']
+        translation=translation.replace("[@àéèíóòú@]. ","")
+    except:
+        printLOG(1,"ERROR Lucy:",sys.exc_info())
+    return(translation)
+  

MTUOC_Marian.py

@@ -0,0 +1,66 @@
+#    MTUOC_Marian
+#    Copyright (C) 2023  Antoni Oliver
+#    v. 07/06/2023
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+import websocket
+import socket
+import time
+import sys
+import config
+
+from MTUOC_misc import printLOG
+
+
+def connect_to_Marian():
+    print("CONNECT TO MARIAN.")
+    from websocket import create_connection
+    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+        conn=s.connect_ex((config.MTEngineIP, config.MTEnginePort))
+        if conn == 0:marianstarted=True
+    service="ws://"+config.MTEngineIP+":"+str(config.MTEnginePort)+"/translate"
+    error=True
+    while error:
+        try:
+            config.ws = create_connection(service)
+            printLOG(1,"Connection with Marian Server created","")
+            error=False
+        except:
+            printLOG(1,"Error: waiting for Marian server to start. Retrying in 2 seconds.",sys.exc_info())
+            time.sleep(2)
+            error=True
+    
+def translate_segment_Marian(segmentPre):
+    translation_candidates={}
+    translation_candidates["segmentNOTAGSPre"]=segmentPre
+    lseg=len(segmentPre)
+    try:
+        config.ws.send(segmentPre)
+    except:
+        printLOG(1,"Error sending segment to Marian.",sys.exc_info())
+    translations = config.ws.recv()
+    tc_aux=translations.split("\n")
+    translation_candidates["translationNOTAGSPre"]=[]
+    translation_candidates["alignments"]=[]
+    for tca in tc_aux:
+        printLOG(3,"TCA:",tca)
+        try:
+            segmentaux=tca.split(" ||| ")[1].strip()
+            alignmentaux=tca.split(" ||| ")[2].strip()
+            translation_candidates["translationNOTAGSPre"].append(segmentaux)
+            translation_candidates["alignments"].append(alignmentaux)
+        except:
+            pass
+    return(translation_candidates)

MTUOC_Moses.py

@@ -0,0 +1,77 @@
+#    MTUOC_Moses
+#    Copyright (C) 2023  Antoni Oliver
+#    v. 07/06/2023
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+import time
+import config
+
+from MTUOC_misc import printLOG
+
+
+def connect_to_Moses():
+    import xmlrpc.client
+    error=True
+    while error:
+        try:
+            proxyMoses = xmlrpc.client.ServerProxy("http://"+config.MTEngineIP+":"+str(config.MTEnginePort)+"/RPC2")
+            error=False
+            printLOG(1,"Connection with Moses Server created","")
+        except:
+            printLOG(1,"Error: waiting for Moses server to start. Retrying in 2 seconds.","")
+            time.sleep(2)
+            error=True
+    return(proxyMoses)
+    
+def translateAliMoses(aliBRUT):
+    newali=[]
+    for a in aliBRUT:
+        at=str(a['source-word'])+"-"+str(a['target-word'])
+        newali.append(at)
+    newali=" ".join(newali)
+
+    return(newali)
+    
+def translate_segment_Moses(segmentPre):
+    param = {"text": segmentPre}
+    result = config.proxyMoses.translate(param)
+    translation_candidates={}
+    translation_candidates["translationNOTAGSPre"]=[]
+    translation_candidates["alignments"]=[]
+    if "nbest" in result:
+        print("NBEST")
+    else:
+        print("UN SOL RESULTAT")
+        translationREP=result['text']
+        alignmentBRUT=result['word-align']
+        alignments=translateAliMoses(alignmentBRUT)
+        translation_candidates["translationNOTAGSPre"].append(translationREP)
+        translation_candidates["alignments"].append(alignments)
+        
+    
+    return(translation_candidates)
+    
+'''
+translation_candidates["translationNOTAGSPre"]=[]
+    translation_candidates["alignments"]=[]
+    for tca in tc_aux:
+        try:
+            segmentaux=tca.split(" ||| ")[1].strip()
+            alignmentaux=tca.split(" ||| ")[2].strip()
+            translation_candidates["translationNOTAGSPre"].append(segmentaux)
+            translation_candidates["alignments"].append(alignmentaux)
+        except:
+            pass
+'''

MTUOC_OpenNMT.py

@@ -0,0 +1,44 @@
+#    MTUOC_OpenNMT
+#    Copyright (C) 2023  Antoni Oliver
+#    v. 07/06/2023
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+import requests
+import time
+import sys
+import config
+
+from MTUOC_misc import printLOG
+
+
+def connect_to_OpenNMT():
+    config.urlOpenNMT = "http://"+config.MTEngineIP+":"+str(config.MTEnginePort)+"/translator/translate"
+    config.headersOpenNMT = {'content-type': 'application/json'}
+    
+    
+def translate_segment_OpenNMT(segmentPre):
+    translation_candidates={}
+    translation_candidates["segmentNOTAGSPre"]=segmentPre
+    params = [{ "src" : segmentPre}]
+    response = requests.post(config.urlOpenNMT, json=params, headers=config.headersOpenNMT)
+    target = response.json()
+    selectedtranslationPre=target[0][0]["tgt"]
+    if "align" in target[0][0]:
+        selectedalignments=target[0][0]["align"][0]
+    else:
+        selectedalignments=""
+    translation_candidates["translationNOTAGSPre"]=[selectedtranslationPre]
+    translation_candidates["alignments"]=[selectedalignments]
+    return(translation_candidates)

MTUOC_misc.py

@@ -0,0 +1,35 @@
+import socket
+from datetime import datetime
+import sys
+
+import config
+
+def printLOG(vlevel,m1,m2=""):
+    cadena=str(datetime.now())+"\t"+str(m1)+"\t"+str(m2)
+    if vlevel<=config.verbosity_level:
+        print(cadena)
+        if config.log_file:
+            config.sortidalog.write(cadena+"\n") 
+
+def get_IP_info(): 
+    try: 
+        host_name = socket.gethostname() 
+        host_ip = socket.gethostbyname(host_name) 
+        s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+        # doesn't even have to be reachable
+        s.connect(('10.255.255.255', 1))
+        IP = s.getsockname()[0]
+        return(IP)
+    except: 
+        IP = '127.0.0.1'
+        return(IP)
+    finally:
+        s.close()
+        
+def capitalizeMTUOC(cadena):
+    resultat=cadena
+    try:
+        resultat=cadena[0].upper()+cadena[1:]
+    except:  
+        pass      
+    return(resultat)

MTUOC_preprocess.py

@@ -0,0 +1,115 @@
+#    MTUOC_preprocess
+#    Copyright (C) 2023  Antoni Oliver
+#    v. 23/05/2023
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+import config
+from MTUOC_misc import printLOG
+import regex as rx
+import re
+import html
+
+    
+def remove_control_characters(cadena):
+    return rx.sub(r'\p{C}', '', cadena)
+    
+re_num = re.compile(r'[\d,\.]+')
+
+def replace_NUMs(segment,code="@NUM@"):
+    trobatsEXPRNUM=re.finditer(re_num,segment)
+    for trobat in trobatsEXPRNUM:
+        if not trobat.group(0) in [".",","]:
+            segment=segment.replace(trobat.group(0),code,1)
+    return(segment)
+    
+def splitnumbers(segment,joiner=""):
+    joiner=joiner+" "
+    xifres = re.findall(re_num,segment)
+    for xifra in xifres:
+        xifrastr=str(xifra)
+        xifrasplit=xifra.split()
+        xifra2=joiner.join(xifra)
+        segment=segment.replace(xifra,xifra2)
+    return(segment)
+
+def preprocess_segment(segment):
+    if config.escape_html_input:
+        segment=html.escape(segment)
+    if config.unescape_html_input:
+        segment=html.unescape(segment)
+    segment=segment.replace(" <tag0>"," <tag0> ")
+    segment=segment.replace(" </tag0>"," </tag0> ")
+    segment=remove_control_characters(segment) 
+    hastags=config.tagrestorer.has_tags(segment)
+    originaltags=config.tagrestorer.get_tags(segment)
+    printLOG(3,"HAS TAGS",hastags)
+    printLOG(3,"TAGS",originaltags)
+    #leading and trailing spaces
+    config.leading_spaces=len(segment)-len(segment.lstrip())
+    config.trailing_spaces=len(segment)-len(segment.rstrip())-1
+    segment=segment.lstrip().rstrip()
+    if config.pre_replace_NUMs:
+        segment=replace_NUMs(segment)
+    if config.pre_split_NUMs:
+        segment=splitnumbers(segment)
+    
+    if config.sentencepiece:
+        try:
+            segmentPre=" ".join(config.spSL.encode(segment))
+        except:
+            printLOG(1,"ERROR preprocess segment:",sys.exc_info())
+    else:
+        segmentPre=segment
+    return(segmentPre)
+    
+def postprocess_segment(segmentPre):
+    try:
+        if config.sentencepiece:
+            segmentPost=config.spTL.decode(segmentPre.split())
+    except:
+            printLOG(1,"ERROR preprocess segment:",sys.exc_info())
+    return(segmentPost)
+
+def tokenizationSL(segment):
+    if config.tokenize_SL and not config.tokenizerSL==None:
+        if config.tokenizerSLType=="MTUOC":
+            tokens=config.tokenizerSL.tokenize(segment)
+        elif config.tokenizerSLType=="Moses":
+            tokens=" ".join(config.tokenizerSL(segment))        
+    else:
+        tokens=segment   
+
+    return(tokens)
+        
+def tokenizationTL(segment):
+    if config.tokenize_TL and not config.tokenizerTL==None:
+        tokens=config.tokenizerTL.tokenize(segment)
+    else:
+        tokens=segment  
+    return(tokens)
+
+def detokenizationSL(tokens):
+    if not config.tokenizerSL==None:
+        segment=config.tokenizerSL.detokenize(tokens)
+    else:
+        segment=tokens   
+    return(tokens)
+        
+def detokenizationTL(tokens):
+    if not config.tokenizerTL==None:
+        segment=config.tokenizerL.detokenize(tokens)
+    else:
+        segment=tokens   
+    return(tokens)

MTUOC_splitnumbers.py

@@ -0,0 +1,44 @@
+#    MTUOC_splitnumbers
+#    Copyright (C) 2023  Antoni Oliver
+#    v. 07/06/2023
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+import re
+import sys
+
+def splitnumbers(segment,joiner="@@"):
+    joiner=joiner+" "
+    xifres = re.findall(re_num,segment)
+    for xifra in xifres:
+        xifrastr=str(xifra)
+        xifrasplit=xifra.split()
+        xifra2=joiner.join(xifra)
+        segment=segment.replace(xifra,xifra2)
+    return(segment)
+
+re_num = re.compile(r'[\d,.\-/]+')
+re_num_tl = re.compile(r'(([\d,.\-/]\s?)+)')
+
+
+if __name__ == "__main__":
+    for line in sys.stdin:
+        line=line.rstrip()
+        output=splitnumbers(line)
+        print(output)
+        
+        
+
+    
+

MTUOC_tags.py

@@ -0,0 +1,426 @@
+#    MTUOC_tags
+#    Copyright (C) 2023  Antoni Oliver
+#    v. 07/06/2023
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+import re
+from collections import Counter
+from bs4 import BeautifulSoup
+
+
+def lreplace(pattern, sub, string):
+    """
+    Replaces 'pattern' in 'string' with 'sub' if 'pattern' starts 'string'.
+    """
+    return re.sub('^%s' % pattern, sub, string)
+
+def rreplace(pattern, sub, string):
+    """
+    Replaces 'pattern' in 'string' with 'sub' if 'pattern' ends 'string'.
+    """
+    return re.sub('%s$' % pattern, sub, string)
+
+class TagRestorer():
+    def __init__(self):
+        self.taglist=["<tag0>","<tag1>","<tag2>","<tag3>","<tag4>","<tag5>","<tag6>","<tag7>","<tag8>","<tag9>","<tag10>","</tag0>","</tag1>","</tag2>","</tag3>","</tag4>","</tag5>","</tag6>","</tag7>","</tag8>","</tag9>","</tag10>"]
+        
+    def has_tags(self, segment):
+        response=False
+        tagsA = re.findall(r'</?.+?/?>', segment)
+        tagsB = re.findall(r'\{[0-9]+\}', segment)
+        if len(tagsA)>0 or len(tagsB)>0:
+            response=True
+        return(response)
+    
+    def get_name(self, tag):
+        name=tag.split(" ")[0].replace("<","").replace(">","").replace("/","")
+        return(name)
+    
+    def get_tags(self, segment):
+        tagsA = re.findall(r'</?.+?/?>', segment)
+        tagsB = re.findall(r'\{[0-9]+\}', segment)
+        tags=tagsA.copy()
+        tags.extend(tagsB)
+        return(tags)
+        
+    def replace_tags(self, segment):
+        equil={}
+        if self.has_tags(segment):
+            tagsA = re.findall(r'</?.+?/?>', segment)
+            tagsB = re.findall(r'\{[0-9]+\}', segment)
+            tags=tagsA.copy()
+            tags.extend(tagsB)
+            conttag=0
+            for tag in tags:
+                if tag.find("</")>-1:
+                    tagrep="</tag"+str(conttag)+">"
+                else:
+                    tagrep="<tag"+str(conttag)+">"
+                segment=segment.replace(tag,tagrep,1)
+                equil[tagrep]=tag
+                if tag in tagsA:
+                    tagclose="</"+self.get_name(tag)+">"
+                    tagcloserep="</tag"+str(conttag)+">"
+                    if segment.find(tagclose)>-1:
+                        segment=segment.replace(tagclose,tagcloserep,1)
+                        equil[tagcloserep]=tagclose
+                        tags.remove(tagclose)
+                conttag+=1
+                
+            return(segment,equil)
+            
+        else:
+            return(segment,equil)
+    
+    def remove_tags(self, segment):
+        segmentnotags=re.sub('(<[^>]+>)', "",segment)
+        segmentnotags=re.sub('({[0-9]+})', "",segmentnotags)
+        segmentnotags=" ".join(segmentnotags.split())
+        return(segmentnotags)
+        
+    def restore_tags_old(self,SOURCENOTAGSTOK, SOURCETAGSTOK, SELECTEDALIGNMENT, TARGETNOTAGSTOK):
+        TARGETTAGLIST=TARGETNOTAGSTOK.split(" ")
+        ali={}
+        for a in SELECTEDALIGNMENT.split():
+            (a1,a2)=a.split("-")
+            a1=int(a1)
+            a2=int(a2)
+            ali[a1]=a2
+        position=0
+        tagpos={}
+        posacu=0
+        SOURCETAGSTOKLIST=SOURCETAGSTOK.split()
+        try:
+            while "▁" in SOURCETAGSTOKLIST: SOURCETAGSTOKLIST.remove("▁")
+        except:
+            pass
+        SOURCENOTAGSTOKLIST=SOURCENOTAGSTOK.split()
+        tagsposition={}
+        cont=0
+        acumulat=0
+        for token in SOURCETAGSTOKLIST:
+            if self.isSTag(token) or self.isSClosingTag(token):
+                if not token in ["<s>","</s>"]:
+                    tagsposition[token]=cont
+                    TARGETTAGLIST.insert(cont,token)
+                    acumulat+=1
+            cont+=1
+        targettags=" ".join(TARGETTAGLIST)
+        return(targettags) 
+            
+    def remove_start_end_tag(self, segment):
+        try:
+            starttag=re.match("(</?tag[0-9]+>)+",segment)
+            starttag=starttag.group()
+        except:            
+            starttag=""
+        try:
+            endtag=re.search("(</?tag[0-9]+>)+$",segment)
+            endtag=endtag.group()
+        except:
+            endtag=""
+        if starttag:
+            segment=lreplace(starttag,"",segment)
+        if endtag:
+            segment=rreplace(endtag,"",segment)
+        return(segment,starttag,endtag)
+        
+    def repairSpacesTags(self,slsegment,tlsegment,delimiters=[" ",".",",",":",";","?","!"]):
+        sltags=self.get_tags(slsegment)
+        tltags=self.get_tags(tlsegment)
+        commontags= list((Counter(sltags) & Counter(tltags)).elements())
+        for tag in commontags:
+            try:
+                tagaux=tag
+                chbfSL=slsegment[slsegment.index(tag)-1]
+                chbfTL=tlsegment[tlsegment.index(tag)-1]
+                tagmod=tag
+                if chbfSL in delimiters and chbfTL not in delimiters:
+                    tagmod=" "+tagmod
+                if not chbfSL in delimiters and chbfTL in delimiters:
+                    tagaux=" "+tagaux
+                try:
+                    chafSL=slsegment[slsegment.index(tag)+len(tag)]
+                except:
+                    pass
+                try:
+                    chafTL=tlsegment[tlsegment.index(tag)+len(tag)]
+                except:
+                    pass
+                if chafSL in delimiters and not chafTL in delimiters:
+                    tagmod=tagmod+" "
+                if not chafSL in delimiters and chafTL in delimiters:
+                    tagaux=tagaux+" "
+                #slsegment=slsegment.replace(tagaux,tagmod,1)
+                tlsegment=tlsegment.replace(tagaux,tagmod,1)
+                tlsegment=tlsegment.replace("  "+tag," "+tag,1)
+                tlsegment=tlsegment.replace(tag+"  ",tag+" ",1)
+                
+
+                
+            except:
+                pass
+        return(tlsegment)
+        
+    def numerate(self,segment):
+        numeratedsegment=[]
+        cont=0
+        for token in segment.split():
+            if not token.replace("▁","").strip() in self.taglist:
+                tokenmod=token+"▂"+str(cont)
+                cont+=1
+            else:
+                tokenmod=token
+            numeratedsegment.append(tokenmod)
+        return(" ".join(numeratedsegment))
+        
+    def retrieve_indexes(self, segment):
+        indexes=[]
+        for token in segment.split():            
+            if token.find("▂")>-1:
+                parts=token.split("▂")
+                try:
+                    index=int(parts[-1])
+                    indexes.append(index)
+                except:
+                    pass
+        if len(indexes)==0:
+            min_value=0
+            max_value=0
+        else:
+            min_value=min(indexes)
+            max_value=max(indexes)
+        return(min_value,max_value)
+        
+    def insert_open_close(self, TARGETTAGSTOKNUM,opentag,closetag,minpos,maxpos):
+        position=0 
+        num=-1
+        opendone=False
+        closedone=False
+        for token in TARGETTAGSTOKNUM:
+            if token.find("▂")>-1:
+                parts=token.split("▂")
+                try:
+                    num=int(parts[-1])
+                except:
+                    num=-1
+            if num==minpos and not opendone:
+                TARGETTAGSTOKNUM.insert(position,opentag)
+                opendone=True
+            elif num==maxpos and not closedone:
+                TARGETTAGSTOKNUM.insert(position+1,closetag)
+                closedone=True
+            position+=1
+        return(TARGETTAGSTOKNUM)   
+    
+    def insert_before(self, segment,insertposition,opentag):
+        position=0
+        num=-1
+        for token in segment:
+            if token.find("▂")>-1:
+                parts=token.split("▂")
+                try:
+                    num=int(parts[-1])
+                except:
+                    num=-1
+            if num==insertposition:
+                segment.insert(position,opentag)
+                break
+            position+=1
+        return(segment)
+        
+    def insert_after(self, segment,insertposition,opentag):
+        position=0
+        num=-1
+        for token in segment:
+            if token.find("▂")>-1:
+                parts=token.split("▂")
+                try:
+                    num=int(parts[-1])
+                except:
+                    num=-1
+            if num==insertposition:
+                segment.insert(position+1,opentag)
+                break
+            position+=1
+        return(segment)
+                
+    
+    def insert_opentag(self, TARGETTAGSTOKNUM, position, opentag):
+        alreadydone=[]
+        position2=0
+        num=-1
+        for token in TARGETTAGSTOKNUM:
+            if token.find("▂")>-1:
+                parts=token.split("▂")
+                try:
+                    num=int(parts[-1])
+                except:
+                    num=-1
+                position2+=1
+            if num==position and not opentag in alreadydone:
+                insertposition=position
+                if insertposition<0: insertposition=0
+                TARGETTAGSTOKNUM=self.insert_before(TARGETTAGSTOKNUM,insertposition,opentag)
+                alreadydone.append(opentag)
+        return(TARGETTAGSTOKNUM)
+                
+    def insert_closingtag(self, TARGETTAGSTOKNUM, position, closingtag):
+        alreadydone=[]
+        position2=0
+        num=-1
+        for token in TARGETTAGSTOKNUM:
+            if token.find("▂")>-1:
+                parts=token.split("▂")
+                try:
+                    num=int(parts[-1])
+                except:
+                    num=-1
+                position2+=1
+            if num==position and not closingtag in alreadydone:
+                insertposition=position
+                if insertposition<0: insertposition=0
+                TARGETTAGSTOKNUM=self.insert_after(TARGETTAGSTOKNUM,insertposition,closingtag)
+                alreadydone.append(closingtag)
+        return(TARGETTAGSTOKNUM)
+                
+                
+                
+                
+        return(TARGETTAGSTOKNUM) 
+
+    def closest_value(self,input_list, input_value):
+        difference = lambda input_list : abs(input_list - input_value)
+        try:
+            res = min(input_list, key=difference)
+        except:
+            res=""
+        return res                             
+        
+    def restore_tags(self,SOURCENOTAGSTOK, SOURCETAGSTOK, SELECTEDALIGNMENT, TARGETNOTAGSTOK):
+        SOURCETAGSTOK=SOURCETAGSTOK.replace(" ▁ "," ")
+        ali={}
+        nmax=0
+        nmin=100000
+        mmax=0
+        mmin=100000
+        for a in SELECTEDALIGNMENT.split():
+            (a1,a2)=a.split("-")
+            a1=int(a1)
+            a2=int(a2)
+            if not a1 in ali: ####AFEGIT AIXÒ
+                ali[a1]=a2
+            if a1>nmax: nmax=a1
+            if a1<nmin: nmin=a1
+            if a2>mmax: mmax=a2
+            if a2<mmin: mmin=a2
+        #chek is all alignments exists
+        nonexisting=[]
+        for i in range(nmin,nmax):
+            try:
+                b=ali[i]
+            except:
+                nonexisting.append(i)
+        inv_ali = {v: k for k, v in ali.items()}
+        inv_nonexisting=[]
+        for i in range(mmin,mmax):
+            try:
+                b=inv_ali[i]
+            except:
+                inv_nonexisting.append(i)
+        for ne in nonexisting:
+            closest=self.closest_value(inv_nonexisting,ne)
+            ali[ne]=closest
+        SOURCENOTAGSTOKNUM=self.numerate(SOURCENOTAGSTOK)
+        SOURCETAGSTOKNUM=self.numerate(SOURCETAGSTOK)
+        TARGETNOTAGSTOKNUM=self.numerate(TARGETNOTAGSTOK)
+        TARGETTAGSTOKNUM=TARGETNOTAGSTOKNUM.split(" ")
+        taglist=self.taglist.copy()
+        #finding open-close pairs
+        for n in range(0,11):
+            opentag="<tag"+str(n)+">"
+            closetag="</tag"+str(n)+">"
+            regexp=opentag+"(.*?)"+closetag
+            trobat=re.findall(regexp, SOURCETAGSTOKNUM, re.DOTALL)
+            if len(trobat)>0 and opentag in taglist and closetag in taglist:
+                (minpos,maxpos)=self.retrieve_indexes(trobat[0])
+                postrad=[]
+                postrad.append(ali[minpos])
+                postrad.append(ali[maxpos])
+                minpostrad=min(postrad)
+                maxpostrad=max(postrad)
+                TARGETTAGSTOKNUM=self.insert_open_close(TARGETTAGSTOKNUM,opentag,closetag,minpostrad,maxpostrad)
+                taglist.remove(opentag)
+                taglist.remove(closetag)
+        #finding open tags
+        for n in range(0,11):
+            opentag="<tag"+str(n)+">"
+            regexp=opentag+" [^\s]+"
+            trobat=re.findall(regexp, SOURCETAGSTOKNUM, re.DOTALL)
+            if len(trobat)>0 and opentag in taglist:
+                posttoken=trobat[0].replace(opentag,"").strip()
+                try:
+                    postnum=int(posttoken.split("▂")[1])
+                except:
+                    postnum=None
+                if not postnum==None and opentag in taglist:
+                    TARGETTAGSTOKNUM=self.insert_opentag(TARGETTAGSTOKNUM, ali[postnum], opentag)
+                    taglist.remove(opentag)
+        #finding closing tags
+        for n in range(0,11):
+            closingtag="</tag"+str(n)+">"
+            regexp="[^\s]+ "+closingtag
+            trobat=re.findall(regexp, SOURCETAGSTOKNUM, re.DOTALL)
+            if len(trobat)>0 and closingtag in taglist:
+                pretoken=trobat[0].replace(closingtag,"").strip()
+                try:
+                    prenum=int(pretoken.split("▂")[1])
+                except:
+                    prenum=None
+                if not prenum==None and closingtag in taglist:
+                    TARGETTAGSTOKNUM=self.insert_closingtag(TARGETTAGSTOKNUM, ali[prenum], closingtag)
+                    taglist.remove(closingtag)
+                
+        #removing numbering        
+        TARGETTAGS=[]
+        for token in TARGETTAGSTOKNUM:
+            TARGETTAGS.append(token.split("▂")[0])
+        
+        TARGETTAGS=" ".join(TARGETTAGS)        
+        return(TARGETTAGS)
+        
+    def fix_xml_tags(self,myxml):
+        if self.has_tags(myxml):
+            tagsPRE=self.get_tags(myxml)
+            myxml2="<fix_xml>"+myxml+"</fix_xml>"
+            soup = BeautifulSoup(myxml2,'xml')
+            fixed=str(soup).replace("<fix_xml>","").replace("</fix_xml>","").split("\n")[-1]
+            tags=self.get_tags(fixed)
+            for TP in tagsPRE:
+                if not TP in tags:
+                    return(myxml)
+            for tag in tags:
+                tag2=tag.replace('"',"'")
+                if myxml.find(tag)==-1 and myxml.find(tag2)==-1:
+                    fixed=fixed.replace(tag,"")
+            
+            if not self.remove_tags(myxml)==self.remove_tags(fixed):
+                fixed=myxml
+        else:
+            fixed=myxml
+        return(fixed)
+                
+                
+            
+        

MTUOC_tokenizer_eng.py

@@ -0,0 +1,360 @@
+#    MTUOC_tokenizer_eng 4.0
+#    Copyright (C) 2024  Antoni Oliver
+#    02/11/2023
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+import string
+import re
+import sys
+import html
+
+QUOTES = (
+#adapted from: https://gist.github.com/goodmami/ quotes.py
+    '\u0022'  # quotation mark (")
+    '\u0027'  # apostrophe (')
+    '\u00ab'  # left-pointing double-angle quotation mark
+    '\u00bb'  # right-pointing double-angle quotation mark
+    '\u2018'  # left single quotation mark
+    '\u2019'  # right single quotation mark
+    '\u201a'  # single low-9 quotation mark
+    '\u201b'  # single high-reversed-9 quotation mark
+    '\u201c'  # left double quotation mark
+    '\u201d'  # right double quotation mark
+    '\u201e'  # double low-9 quotation mark
+    '\u201f'  # double high-reversed-9 quotation mark
+    '\u2039'  # single left-pointing angle quotation mark
+    '\u203a'  # single right-pointing angle quotation mark
+    '\u300c'  # left corner bracket
+    '\u300d'  # right corner bracket
+    '\u300e'  # left white corner bracket
+    '\u300f'  # right white corner bracket
+    '\u301d'  # reversed double prime quotation mark
+    '\u301e'  # double prime quotation mark
+    '\u301f'  # low double prime quotation mark
+    '\ufe41'  # presentation form for vertical left corner bracket
+    '\ufe42'  # presentation form for vertical right corner bracket
+    '\ufe43'  # presentation form for vertical left corner white bracket
+    '\ufe44'  # presentation form for vertical right corner white bracket
+    '\uff02'  # fullwidth quotation mark
+    '\uff07'  # fullwidth apostrophe
+    '\uff62'  # halfwidth left corner bracket
+    '\uff63'  # halfwidth right corner bracket
+)
+
+HYPENS = ('\u2010','\u2011','\u2012','\u2013')
+
+class Tokenizer():
+    def __init__(self):
+        self.specialchars=["¿","¡",]
+        self.specialchars.extend(QUOTES)
+        self.specialchars.extend(HYPENS)
+        self.subs=["￭'s","￭'ll","￭'t","￭'cause","￭'d","￭'em","￭'ve","￭'dn","￭'m","￭'n","￭'re","￭'til","￭'tween","￭'all","ol'￭"]
+        self.re_num = re.compile(r'[\d\,\.]+')
+        self.abr=["a.a.","a.a.a.","a.a.u.","a.b.","a.b.a","abbr.","abr.","a.c.","acad.","aclu.","a.d.","adm.","a.e.c.","a.f.l.","afl-cio","afrik.","a.i.a.","a.k.c.","a.l.","a.l.a.","ala.","alt.","alta.","a.m.","a.m.a.","a.m.p.","a.m.u.","antilog.","a.p.","arab.","ariz.","ark.","a.s.","ascap.","at.no.","at.wt.","a.u.","aug.","a.v.","avdp.","ave.","b.a.","b.b.c.","b.c.","b.d.","b.lit.","b.mus.","b.p.","brig.gen.","b.s.","b.t.u.","bul.","bulg.","c.","cal.","calif.","cant.","capt.","c.c.","c.d.","cent.","cento.","c.e.o.","c.g.s.","chem.","chin.","chron.","cie.","c.i.a.","c.i.d.","c.i.o.","cl.","c.m.","cn.","co.","col.","coll.","colo.","comdr.","comp.","com.pop.","conn.","cor.","corp.","cos.","cot.","coul.","cp.","c.p.a.","c.p.l.","c.p.o.","c.s.c.","c.u.","dan.","dar.","d.c.","d.c.l.","d.d.","d.d.s.","d.d.t.","dec.","del.","dept.","deut.","dist.","div.","dr.","ds.","d.sc.","du.","e.c.","e.c.a.","eccles.","ecclus.","ed.","e.d.c.","e.e.","e.e.a.","e.e.c.","e.e.o.c.","e.f.t.a.","e.g.","e.m.f.","e.m.u.","eng.","enl.","eph.","e.r.a.","e.r.p.","e.s.c.","esp.","est.","e.u.","ev.","ex.","ezek.","f.a.a.","fac.","f.a.o.","f.b.i.","f.c.c.","f.d.a.","feb.","f.e.p.c.","finn.","fl.","fla.","floz.","f.m.","fr.","ft.","f.t.c.","g.","ga.","gal.","gall.","gatt.","g.d.p.","gen.","ger.","gm.","g.m.b.","g.m.t.","g.n.p.","g.o.p.","gov.","gr.","grad.","grm.","hab.","hag.","heb.","h.m.s.","hon.","hr.","hung.","hz.","i.a.u.","i.b.m.","i.b.t.","i.c.a.o.","i.c.b.m.","i.c.c.","icel.","i.e.","i.g.y.","ilgwu.","ill.","i.l.o.","i.m.f.","inc.","incl.","ind.","ing.","inst.","introd.","i.q.","i.r.a.","i.r.b.m.","i.r.s.","isa.","ital.","i.t.u.","i.u.p.a.c.","i.w.w.","jan.","jap.","j.d.","jer.","j.g.","jr.","j.","kc.","kg.","kgb.","kgm.","k.k.k.","kl.","km.","kw.","kwh.","ky.","l.","la.","lam.","lat.","lb.","lev.","l.h.d.","lib.","lith.","litt.b.","litt.d.","ll.b.","ll.d.","l.s.d.","lt.","lt.col.","ltd.","lt.gen.","lt.gov.","lts.","m.a.","mac.","maj.","maj.gen.","mal.","mass.","mass.no.","m.b.","m.d.","md.","m.e.","messrs.","mev.","mex.","mfg.","mg.","m.h.g.","mi.","mich.","min.","minn.","miss.","mks.","ml.","mlle.","mls.","mm.","mme.","mo.","mont.","m.p.","m.p.h.","mph.","mr.","mrs.","m.s.","ms.","msgr.","mss.","mt.","mts.","mus.","mus.b.","mus.d.","n.a.a.c.p.","n.a.f.t.a.","n.a.s.a.","n.a.s.d.a.q.","n.a.t.o.","n.b.","n.b.a.","n.c.","n.c.a.a.","n.c.o.","n.dak.","n.e.","n.e.a.","nebr.","neh.","nev.","n.f.","n.f.l.","n.h.","n.h.l.","n.j.","nl.","n.l.r.b.","n.mex.","nnw","no.","non-u.s.","nor.","nov.","n.r.a.","n.r.c.","n.s.","n.s.f.","num.","n.v.","n.y.","n.y.a.","n.y.s.e.","o.a.s.","obad.","oct.","o.e.","o.e.c.d.","o.e.o.","o.e.s.","o.fr.","o.h.g.","okla.","o.n.","ont.","op.","o.p.a.","o.s.","o.s.c.e.","o.s.s.","o.z.","oz.","pa.","p.a.u.","pd.d.","p.e.i.","pers.","p.f.c.","p.g.a.","ph.","ph.b.","ph.d.","philip.","pk.","pkg.","pl.","plc","p.m","p.m.","pn.","po.","pol.","pop.","port.","prof.","prov.","prov(s).","ps.","pseud.","pss.","pt.","pts.","pub.","pvt.","p.w.a.","q.t.","qt.","qts.","que.","r.a.","r.a.f.","rep.","reps.","repr.","rev.","r.i.","r.n.","r.n.a.","rom.","r.o.t.c.","r.p.m.","rpm.","r.r.","r.s.f.s.r.","r.s.v.","rt.rev.","rus.","r.v.","sam.","sask.","s.c.","sc.d.","s.dak.","s.e.","s.e.a.t.o.","sec.","sen.","sens.","sept.","ser.","sgt.","s.j.","skt.","sl.","s.o.s.","span.","s.p.a.","s.p.c.a.","s.p.c.c.","sp.gr.","s.q.","sr.","s.s.","s.s.r.","st.","s.t.d.","s.t.e.","s.t.p.","s.w.","swed.","t.","tablesp.","t.a.n.","t.a.s.s.","tb.","tbl.","tble.","tbles.","tbls.","tblsp.","tbs.","tbsn.","tbsp.","tbsps.","teas.","tenn.","thess.","tim.","t.n.t","tr.","ts.","tsp.","tsps.","turk.","t.v.a.","u.a.w.","u.h.f.","ukr.","u.m.w.","u.n.","uninc.","univ.","u.n.r.r.a.","u.p.i.","u.s.","u.s.a.","u.s.a.f.","u.s.c.g.","u.s.m.c.","u.s.n.","u.s.o.","u.s.s.","u.s.s.r.","u.t.","va.","var.","ved.","v.","v.f.w.","v.h.f.","vol.","vs.","vt.","w.a.c.","w.c.t.u.","w.e.u.","w.f.t.u.","wis.","wmo.","wpa.","wt.","wto.","w.va.","wyo.","yd.","y.m.c.a.","y.m.h.a.","y.w.c.a.","y.w.h.a.","zech.","zeph."]
+        abr_aux=[]
+        abr_aux=[]
+        for a in self.abr:
+            am1=a.capitalize()
+            am2=a.upper()
+            abr_aux.append(am1)
+            abr_aux.append(am2)
+        self.abr.extend(abr_aux)
+        self.setabr=set(self.abr)
+        self.abrsig=()
+    def split_numbers(self,segment):
+        xifres = re.findall(self.re_num,segment)
+        for xifra in xifres:
+            xifrastr=str(xifra)
+            xifrasplit=xifra.split()
+            xifra2=[]
+            contpos=0
+            for x in xifrastr:
+                if not contpos==0: xifra2.append(" ￭")
+                xifra2.append(x)
+                contpos+=1
+            xifra2="".join(xifra2)
+            segment=segment.replace(xifra,xifra2)
+        return(segment)
+
+
+
+    def protect_tags(self, segment):
+        tags=re.findall(r'<[^>]+>',segment)
+        for tag in tags:
+            ep=False
+            ef=False
+            if segment.find(" "+tag)==-1:ep=True
+            if segment.find(tag+" ")==-1:ef=True
+            tagmod=tag.replace("<","&#60;").replace(">","&#62;").replace("=","&#61;").replace("'","&#39;").replace('"',"&#34;").replace("/","&#47;").replace(" ","&#32;")
+            if ep: tagmod=" ￭"+tagmod
+            if ef: tagmod=tagmod+"￭ "
+            segment=segment.replace(tag,tagmod)
+        tags=re.findall(r'\{[0-9]+\}',segment)
+        for tag in tags:
+            ep=False
+            ef=False
+            if segment.find(" "+tag)==-1:ep=True
+            if segment.find(tag+" ")==-1:ef=True
+            tagmod=tag.replace("{","&#123;").replace("}","&#125;")
+            if ep: tagmod=" ￭"+tagmod
+            if ef: tagmod=tagmod+"￭ "
+            segment=segment.replace(tag,tagmod)
+        return(segment) 
+    
+    def protect_abr(self,cadena):
+        setcadena=set(cadena.split(" "))
+        common=(self.setabr & setcadena)
+        self.abrsig=list(common)
+        for a in common:
+            amod=a+" "
+            apro=a.replace(".","&#46;")
+            cadena=cadena.replace(amod,apro)
+        sigles=re.findall(r'((\w\.){2,})',cadena)
+        for s in sigles:
+            a=s[0]
+            self.abrsig.append(a)
+            amod=a+" "
+            apro=a.replace(".","&#46;")
+            cadena=cadena.replace(amod,apro)
+        return(cadena)
+    
+    def unprotect(self, cadena):
+        cadena=cadena.replace("&#39;","'").replace("&#45;","-").replace("&#60;","<").replace("&#62;",">").replace("&#34;",'"').replace("&#61;","=").replace("&#32;","▁").replace("&#47;","/").replace("&#123;","{").replace("&#125;","}")
+        return(cadena)
+    
+    def unprotect_tags(self, cadena):
+        cadena=cadena.replace("&#60;","<").replace("&#62;",">")
+        return(cadena)
+    
+    def unprotect_abr(self,cadena):
+        for a in self.abrsig:
+            amod=a+" "
+            apro=a.replace(".","&#46;")
+            cadena=cadena.replace(apro,amod)
+        return(cadena)
+
+    def main_tokenizer(self,segment):
+        segment=" "+segment+" "
+        cadena=self.protect_tags(segment)
+        cadena=self.protect_abr(segment)
+        
+        for s in [".",","]:
+            pass
+        for s in self.subs:
+            sA=s.replace("￭","")
+            sB=s.replace("'","&#39;").replace("-","&#45;")
+            if s.startswith("￭"):sB=" "+sB
+            if s.endswith("￭"):sB=sB+" "
+            cadena = re.sub(sA+r'\b', sB, cadena)
+            cadena = re.sub(r'\b'+sA, sB, cadena)
+            cadena = re.sub(sA.upper()+r'\b', sB.upper(), cadena)
+            cadena = re.sub(r'\b'+sA.upper(), sB.upper(), cadena)
+        punt=list(string.punctuation)
+        exceptions=["&",";","#"]
+        for e in exceptions:
+            punt.remove(e)
+        punt.remove(".")
+        punt.remove(",")        
+        cadena = re.sub(r'\.(\D)'," . "+r"\1", cadena)
+        cadena = re.sub(r'\,(\D)'," , "+r"\1", cadena)
+        for p in punt:
+            ppre=" ￭"+p
+            ppost=p+"￭ "
+            try:
+                expr1="(\\S)\\"+p+"(\\s)"
+                expr2=r"\1"+ppre+r"\2"
+                cadena = re.sub(expr1,expr2, cadena)
+                expr1="(\\s)\\"+p+"(\\S)"
+                expr2=r"\1"+ppost+r"\2"
+                cadena = re.sub(expr1,expr2, cadena)
+            except:
+                pass
+        cadena=self.unprotect_tags(cadena)
+        cadena=self.unprotect_abr(cadena)
+        for p in self.specialchars:
+            pmod=p+" "
+            if cadena.find(pmod)>=-1:
+                pmod2=p+"￭ "
+                cadena=cadena.replace(p,pmod2)
+            pmod=" "+p
+            if cadena.find(pmod)>=-1:
+                pmod2=" ￭"+p
+                cadena=cadena.replace(p,pmod2)
+        
+        
+        
+        
+        cadena=self.unprotect(cadena)
+        
+        for p in exceptions:
+            pmod=p+" "
+            if cadena.find(pmod)>=-1:
+                pmod2=p+"￭ "
+                cadena=cadena.replace(p,pmod2)
+            pmod=" "+p
+            if cadena.find(pmod)>=-1:
+                pmod2=" ￭"+p
+                cadena=cadena.replace(p,pmod2)    
+        
+        cadena=cadena.replace("▁"," ")
+        cadena=' '.join(cadena.split())   
+        return(cadena)
+
+    def tokenize(self,segment):
+        tokenized=self.main_tokenizer(segment)
+        tokenized=tokenized.replace("￭","")
+        tokenized=' '.join(tokenized.split()) 
+        return(tokenized)
+        
+    def detokenize(self, segment):
+        for sub in self.subs:
+            subA=sub.replace("￭"," ")
+            subB=sub.replace("￭","")
+            segment=segment.replace(subA,subB)
+            segment=segment.replace(subA.capitalize(),subB.capitalize())
+            segment=segment.replace(subA.upper(),subB.upper())
+        segment=segment.replace(" .",".")
+        segment=segment.replace(" ,",",")
+        segment=segment.replace(" :",":")
+        segment=segment.replace(" ;",";")
+        segment=segment.replace(" :",":")
+        segment=segment.replace(" )",")")
+        segment=segment.replace("( ","(")
+        segment=segment.replace(" ]","]")
+        segment=segment.replace("[ ","[")
+        segment=segment.replace(" }","}")
+        segment=segment.replace("{ ","{")
+        segment=segment.replace(" !","!")
+        segment=segment.replace("¡ ","¡")
+        segment=segment.replace(" ?","?")
+        segment=segment.replace("¿ ","¿")
+        segment=segment.replace(" »","»")
+        segment=segment.replace("« ","«")
+        segment=segment.replace("‘ ","‘")
+        segment=segment.replace(" ’","’")
+        segment=segment.replace("“ ","“")
+        segment=segment.replace(" ”","”")
+        segment=' '.join(segment.split()) 
+        return(segment)
+
+    def tokenize_j(self,segment):
+        tokenized=self.main_tokenizer(segment)
+        tokenized=' '.join(tokenized.split()) 
+        return(tokenized)
+
+    def detokenize_j(self,segment):
+        segment=segment.replace(" ￭","")
+        segment=segment.replace("￭ ","")
+        segment=segment.replace("￭","")
+        detok=segment
+        detok=' '.join(detok.split()) 
+        return(detok)
+        
+    def tokenize_jn(self,segment):
+        tokenized=self.tokenize_j(segment)
+        tokenized=self.split_numbers(tokenized)
+        tokenized=' '.join(tokenized.split()) 
+        return(tokenized)
+
+    def detokenize_jn(self,segment):
+        segment=self.detokenize_j(segment)
+        return(segment)
+        
+    def tokenize_s(self,segment):
+        tokenized=self.main_tokenizer(segment)
+        tokenized=tokenized.replace("￭ ","￭")
+        tokenized=tokenized.replace(" ￭","￭")
+        tokenized=tokenized.replace(" "," ▁")
+        tokenized=tokenized.replace("￭"," ")
+        tokenized=' '.join(tokenized.split()) 
+        return(tokenized)
+        
+    def detokenize_s(self,segment):
+        segment=segment.replace(" ","")
+        segment=segment.replace("▁"," ")
+        detok=segment
+        detok=' '.join(detok.split()) 
+        return(detok)
+
+    def tokenize_sn(self,segment):
+        tokenized=self.main_tokenizer(segment)
+        tokenized=self.split_numbers(tokenized)
+        tokenized=tokenized.replace("￭ ","￭")
+        tokenized=tokenized.replace(" ￭","￭")
+        tokenized=tokenized.replace(" "," ▁")
+        tokenized=tokenized.replace("￭"," ")
+        tokenized=' '.join(tokenized.split()) 
+        return(tokenized)
+
+    def detokenize_sn(self,segment):
+        segment=self.detokenize_s(segment)
+        return(segment)        
+    
+def print_help():
+    print("MTUOC_tokenizer_cat.py A tokenizer for Catalan, usage:")
+    print("Simple tokenization:")
+    print('    cat "sentence to tokenize." | python3 MTUOC_tokenizer_cat.py tokenize')
+    print('    python3 MTUOC_tokenizer_cat.py tokenize < file_to_tokenize > tokenized_file')
+    print()
+    print("Simple detokenization:")
+    print('    cat "sentence to tokenize." | python3 MTUOC_tokenizer_cat.py detokenize')
+    print('    python3 MTUOC_tokenizer_cat.py detokenize < file_to_detokenize > detokenized_file')
+    print()
+    print("Advanced options:")
+    print("    tokenization/detokenization with joiner marks (￭): tokenize_j / detokenize_j")
+    print("    tokenization/detokenization with joiner marks (￭) and number splitting: tokenize_jn / detokenize_jn")
+    print("    tokenization/detokenization with splitter marks (▁): tokenize_s / detokenize_s")
+    print("    tokenization/detokenization with splitter marks (▁) and number splitting: tokenize_sn / detokenize_sn")
+        
+
+if __name__ == "__main__":
+    if len(sys.argv)>1:
+        if sys.argv[1]=="-h" or sys.argv[1]=="--help":
+            print_help()
+            sys.exit()
+        action=sys.argv[1]
+    else:
+        action="tokenize"
+    tokenizer=Tokenizer()
+    for line in sys.stdin:
+        line=line.strip()
+        #normalization of apostrophe
+        line=line.replace("’","'")
+        line=html.unescape(line)
+        if action=="tokenize":
+            outsegment=tokenizer.tokenize(line)
+        elif action=="detokenize":
+            outsegment=tokenizer.detokenize(line)
+        
+        elif action=="tokenize_s":
+            outsegment=tokenizer.tokenize_s(line)
+        elif action=="detokenize_s":
+            outsegment=tokenizer.detokenize_s(line)
+        elif action=="tokenize_sn":
+            outsegment=tokenizer.tokenize_sn(line)
+        elif action=="detokenize_sn":
+            outsegment=tokenizer.detokenize_sn(line)
+        
+        elif action=="tokenize_j":
+            outsegment=tokenizer.tokenize_j(line)
+        elif action=="detokenize_j":
+            outsegment=tokenizer.detokenize_j(line)
+        elif action=="tokenize_jn":
+            outsegment=tokenizer.tokenize_jn(line)
+        elif action=="detokenize_jn":
+            outsegment=tokenizer.detokenize_jn(line)
+        
+        print(outsegment)
+        

MTUOC_tokenizer_spa.py

@@ -0,0 +1,359 @@
+#    MTUOC_tokenizer_spa 4.0
+#    Copyright (C) 2024  Antoni Oliver
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+import string
+import re
+import sys
+import html
+
+QUOTES = (
+#adapted from: https://gist.github.com/goodmami/ quotes.py
+    '\u0022'  # quotation mark (")
+    '\u0027'  # apostrophe (')
+    '\u00ab'  # left-pointing double-angle quotation mark
+    '\u00bb'  # right-pointing double-angle quotation mark
+    '\u2018'  # left single quotation mark
+    '\u2019'  # right single quotation mark
+    '\u201a'  # single low-9 quotation mark
+    '\u201b'  # single high-reversed-9 quotation mark
+    '\u201c'  # left double quotation mark
+    '\u201d'  # right double quotation mark
+    '\u201e'  # double low-9 quotation mark
+    '\u201f'  # double high-reversed-9 quotation mark
+    '\u2039'  # single left-pointing angle quotation mark
+    '\u203a'  # single right-pointing angle quotation mark
+    '\u300c'  # left corner bracket
+    '\u300d'  # right corner bracket
+    '\u300e'  # left white corner bracket
+    '\u300f'  # right white corner bracket
+    '\u301d'  # reversed double prime quotation mark
+    '\u301e'  # double prime quotation mark
+    '\u301f'  # low double prime quotation mark
+    '\ufe41'  # presentation form for vertical left corner bracket
+    '\ufe42'  # presentation form for vertical right corner bracket
+    '\ufe43'  # presentation form for vertical left corner white bracket
+    '\ufe44'  # presentation form for vertical right corner white bracket
+    '\uff02'  # fullwidth quotation mark
+    '\uff07'  # fullwidth apostrophe
+    '\uff62'  # halfwidth left corner bracket
+    '\uff63'  # halfwidth right corner bracket
+)
+
+HYPENS = ('\u2010','\u2011','\u2012','\u2013')
+
+class Tokenizer():
+    def __init__(self):
+        self.specialchars=["¿","¡",]
+        self.specialchars.extend(QUOTES)
+        self.specialchars.extend(HYPENS)
+        self.subs=[]
+        self.re_num = re.compile(r'[\d\,\.]+')
+        self.abr=["aa.rr.","abr.","abrev.","a.c.","adj.","adm.","admón.","afma.","afmas.","afmo.","afmos.","ag.","ago.","am.","ap.","apdo.","art.","arts.","arz.","arzbpo.","assn.","atte.","av.","avda.","bros.","bv.","cap.","caps.","cg.","cgo.","cia.","cía.","cit.","co.","col.","corp.","cos.","cta.","cte.","ctra.","cts.","d.c.","dcha.","dept.","depto.","dic.","doc.","docs.","dpt.","dpto.","dr.","dra.","dras.","dres.","dto.","dupdo.","ed.","ee.uu.","ej.","emma.","emmas.","emmo.","emmos.","ene.","entlo.","entpo.","esp.","etc.","ex.","excm.","excma.","excmas.","excmo.","excmos.","fasc.","fdo.","feb.","fig.","figs.","fol.","fra.","gb.","gral.","hnos.","hros.","ib.","ibid.","ibíd.","id.","íd.","ilm.","ilma.","ilmas.","ilmo.","ilmos.","iltre.","inc.","intr.","ít.","izq.","izqda.","izqdo.","jr.","jul.","jun.","lám.","lda.","ldo.","lib.","lim.","loc.","ltd.","ltda.","mar.","máx.","may.","mín.","mons.","mr.","mrs.","ms.","mss.","mtro.","nov.","ntra.","ntro.","núm.","ob.","obpo.","oct.","op.","pág.","págs.","párr.","pd.","ph.","pje.","pl.","plc.","pm.","pp.","ppal.","pral.","prof.","pról.","prov.","ps.","pta.","ptas.","pte.","pts.","pza.","rda.","rdo.","ref.","reg.","rel.","rev.","revda.","revdo.","rma.","rmo.","r.p.m.","rte.","sdad.","sec.","secret.","sep.","sig.","smo.","sr.","sra.","sras.","sres.","srs.","srta.","ss.mm.","sta.","sto.","sust.","tech.","tel.","teléf.","telf.","ten.","tfono.","tít.","tlf.","ud.","uds.","vda.","vdo.","vid.","vol.","vols.","vra.","vro.","vta.","abr.","adj.","admón.","admor.","admora.","admtvo.","admvo.","advo.","ag.","ago.","agt.","a.m.","ap.","apdo.","apénd.","aprox.","aptdo.","art.","asoc.","att.","av.","avda.","ayto.","bach.","bibl.","bol.","calif.","cast.","cat.","c.c.","cc.oo.","cf.","cfr.","cjón.","cta.","ctra.","d.","dª.","dcha.","d.e.p.","depto.","dic.","dicbre.","dir.","disp.","dña.","do.","doc.","dom.","dr.","dra.","dto.","dupl.","e.","ed.","ej.","emmo.","ene.","entlo.","epitomadamente.","esc.","esp.","etc.","excma.","excmo.","exp.","expte.","ext.","fac.","facs.","fasc.","fdo.","feb.","febr.","fig.","fra.","gob.","gral.","hble.","hnos.","hosp.","ib.","ibid.","ibíd.","id.","íd.","il.","ilma.","ilmo.","iltre.","inc.","intr.","izq.","izqda.","j.","j.c.","ju.","jue.","jul.","jun.","k.o.","lcdo.","lda.","ldo.","lic.","ltda.","lu.","lun.","ma.","mar.","mart.","máx.","may.","mi.","mié.","miérc.","min.","mín.","mn.","mons.","my.","mzo.","n.b.","nº.","nov.","novbre.","nro.","n.s.","ntra.","ntro.","núm.","núms.","oct.","op.","pág.","pár.","párr.","p.ej.","pl.","p.m.","p.o.","pp.","pp.","pral.","proc.","prof.","ps.","p.s.","pte.","pts.","p.v.p.","pza.","q.d.e.p.","q.e.g.e.","r.d.","r.d.l.","rdo.","ref.","r.i.p.","r.o.","r.p.m.","rvdo.","s.","s.a.","sa.","s.a.","sáb.","sdad.","sep.","sept.","setbre.","s.l.","s.l.u.","s.m.","sr.","sra.","sras.","sres.","s.s.","ss.mm.","ssp.","stmo.","subsp.","tel.","trad.","ud.","ud.","udes.","uds.","urb.","v.a.","v.a.r.","vd.","vda.","vds.","vi.","vid.","vie.","vnes.","vol."]
+        abr_aux=[]
+        abr_aux=[]
+        for a in self.abr:
+            am1=a.capitalize()
+            am2=a.upper()
+            abr_aux.append(am1)
+            abr_aux.append(am2)
+        self.abr.extend(abr_aux)
+        self.setabr=set(self.abr)
+        self.abrsig=()
+    def split_numbers(self,segment):
+        xifres = re.findall(self.re_num,segment)
+        for xifra in xifres:
+            xifrastr=str(xifra)
+            xifrasplit=xifra.split()
+            xifra2=[]
+            contpos=0
+            for x in xifrastr:
+                if not contpos==0: xifra2.append(" ￭")
+                xifra2.append(x)
+                contpos+=1
+            xifra2="".join(xifra2)
+            segment=segment.replace(xifra,xifra2)
+        return(segment)
+
+
+
+    def protect_tags(self, segment):
+        tags=re.findall(r'<[^>]+>',segment)
+        for tag in tags:
+            ep=False
+            ef=False
+            if segment.find(" "+tag)==-1:ep=True
+            if segment.find(tag+" ")==-1:ef=True
+            tagmod=tag.replace("<","&#60;").replace(">","&#62;").replace("=","&#61;").replace("'","&#39;").replace('"',"&#34;").replace("/","&#47;").replace(" ","&#32;")
+            if ep: tagmod=" ￭"+tagmod
+            if ef: tagmod=tagmod+"￭ "
+            segment=segment.replace(tag,tagmod)
+        tags=re.findall(r'\{[0-9]+\}',segment)
+        for tag in tags:
+            ep=False
+            ef=False
+            if segment.find(" "+tag)==-1:ep=True
+            if segment.find(tag+" ")==-1:ef=True
+            tagmod=tag.replace("{","&#123;").replace("}","&#125;")
+            if ep: tagmod=" ￭"+tagmod
+            if ef: tagmod=tagmod+"￭ "
+            segment=segment.replace(tag,tagmod)
+        return(segment) 
+    
+    def protect_abr(self,cadena):
+        setcadena=set(cadena.split(" "))
+        common=(self.setabr & setcadena)
+        self.abrsig=list(common)
+        for a in common:
+            amod=a+" "
+            apro=a.replace(".","&#46;")
+            cadena=cadena.replace(amod,apro)
+        sigles=re.findall(r'((\w\.){2,})',cadena)
+        for s in sigles:
+            a=s[0]
+            self.abrsig.append(a)
+            amod=a+" "
+            apro=a.replace(".","&#46;")
+            cadena=cadena.replace(amod,apro)
+        return(cadena)
+    
+    def unprotect(self, cadena):
+        cadena=cadena.replace("&#39;","'").replace("&#45;","-").replace("&#60;","<").replace("&#62;",">").replace("&#34;",'"').replace("&#61;","=").replace("&#32;","▁").replace("&#47;","/").replace("&#123;","{").replace("&#125;","}")
+        return(cadena)
+    
+    def unprotect_tags(self, cadena):
+        cadena=cadena.replace("&#60;","<").replace("&#62;",">")
+        return(cadena)
+    
+    def unprotect_abr(self,cadena):
+        for a in self.abrsig:
+            amod=a+" "
+            apro=a.replace(".","&#46;")
+            cadena=cadena.replace(apro,amod)
+        return(cadena)
+
+    def main_tokenizer(self,segment):
+        segment=" "+segment+" "
+        cadena=self.protect_tags(segment)
+        cadena=self.protect_abr(segment)
+        
+        for s in [".",","]:
+            pass
+        for s in self.subs:
+            sA=s.replace("￭","")
+            sB=s.replace("'","&#39;").replace("-","&#45;")
+            if s.startswith("￭"):sB=" "+sB
+            if s.endswith("￭"):sB=sB+" "
+            cadena = re.sub(sA+r'\b', sB, cadena)
+            cadena = re.sub(r'\b'+sA, sB, cadena)
+            cadena = re.sub(sA.upper()+r'\b', sB.upper(), cadena)
+            cadena = re.sub(r'\b'+sA.upper(), sB.upper(), cadena)
+        punt=list(string.punctuation)
+        exceptions=["&",";","#"]
+        for e in exceptions:
+            punt.remove(e)
+        punt.remove(".")
+        punt.remove(",")        
+        cadena = re.sub(r'\.(\D)'," . "+r"\1", cadena)
+        cadena = re.sub(r'\,(\D)'," , "+r"\1", cadena)
+        for p in punt:
+            ppre=" ￭"+p
+            ppost=p+"￭ "
+            try:
+                expr1="(\\S)\\"+p+"(\\s)"
+                expr2=r"\1"+ppre+r"\2"
+                cadena = re.sub(expr1,expr2, cadena)
+                expr1="(\\s)\\"+p+"(\\S)"
+                expr2=r"\1"+ppost+r"\2"
+                cadena = re.sub(expr1,expr2, cadena)
+            except:
+                pass
+        cadena=self.unprotect_tags(cadena)
+        cadena=self.unprotect_abr(cadena)
+        for p in self.specialchars:
+            pmod=p+" "
+            if cadena.find(pmod)>=-1:
+                pmod2=p+"￭ "
+                cadena=cadena.replace(p,pmod2)
+            pmod=" "+p
+            if cadena.find(pmod)>=-1:
+                pmod2=" ￭"+p
+                cadena=cadena.replace(p,pmod2)
+        
+        
+        
+        
+        cadena=self.unprotect(cadena)
+        
+        for p in exceptions:
+            pmod=p+" "
+            if cadena.find(pmod)>=-1:
+                pmod2=p+"￭ "
+                cadena=cadena.replace(p,pmod2)
+            pmod=" "+p
+            if cadena.find(pmod)>=-1:
+                pmod2=" ￭"+p
+                cadena=cadena.replace(p,pmod2)    
+        
+        cadena=cadena.replace("▁"," ")
+        cadena=' '.join(cadena.split())   
+        return(cadena)
+
+    def tokenize(self,segment):
+        tokenized=self.main_tokenizer(segment)
+        tokenized=tokenized.replace("￭","")
+        tokenized=' '.join(tokenized.split()) 
+        return(tokenized)
+        
+    def detokenize(self, segment):
+        for sub in self.subs:
+            subA=sub.replace("￭"," ")
+            subB=sub.replace("￭","")
+            segment=segment.replace(subA,subB)
+            segment=segment.replace(subA.capitalize(),subB.capitalize())
+            segment=segment.replace(subA.upper(),subB.upper())
+        segment=segment.replace(" .",".")
+        segment=segment.replace(" ,",",")
+        segment=segment.replace(" :",":")
+        segment=segment.replace(" ;",";")
+        segment=segment.replace(" :",":")
+        segment=segment.replace(" )",")")
+        segment=segment.replace("( ","(")
+        segment=segment.replace(" ]","]")
+        segment=segment.replace("[ ","[")
+        segment=segment.replace(" }","}")
+        segment=segment.replace("{ ","{")
+        segment=segment.replace(" !","!")
+        segment=segment.replace("¡ ","¡")
+        segment=segment.replace(" ?","?")
+        segment=segment.replace("¿ ","¿")
+        segment=segment.replace(" »","»")
+        segment=segment.replace("« ","«")
+        segment=segment.replace("‘ ","‘")
+        segment=segment.replace(" ’","’")
+        segment=segment.replace("“ ","“")
+        segment=segment.replace(" ”","”")
+        segment=' '.join(segment.split()) 
+        return(segment)
+
+    def tokenize_j(self,segment):
+        tokenized=self.main_tokenizer(segment)
+        tokenized=' '.join(tokenized.split()) 
+        return(tokenized)
+
+    def detokenize_j(self,segment):
+        segment=segment.replace(" ￭","")
+        segment=segment.replace("￭ ","")
+        segment=segment.replace("￭","")
+        detok=segment
+        detok=' '.join(detok.split()) 
+        return(detok)
+        
+    def tokenize_jn(self,segment):
+        tokenized=self.tokenize_j(segment)
+        tokenized=self.split_numbers(tokenized)
+        tokenized=' '.join(tokenized.split()) 
+        return(tokenized)
+
+    def detokenize_jn(self,segment):
+        segment=self.detokenize_j(segment)
+        return(segment)
+        
+    def tokenize_s(self,segment):
+        tokenized=self.main_tokenizer(segment)
+        tokenized=tokenized.replace("￭ ","￭")
+        tokenized=tokenized.replace(" ￭","￭")
+        tokenized=tokenized.replace(" "," ▁")
+        tokenized=tokenized.replace("￭"," ")
+        tokenized=' '.join(tokenized.split()) 
+        return(tokenized)
+        
+    def detokenize_s(self,segment):
+        segment=segment.replace(" ","")
+        segment=segment.replace("▁"," ")
+        detok=segment
+        detok=' '.join(detok.split()) 
+        return(detok)
+
+    def tokenize_sn(self,segment):
+        tokenized=self.main_tokenizer(segment)
+        tokenized=self.split_numbers(tokenized)
+        tokenized=tokenized.replace("￭ ","￭")
+        tokenized=tokenized.replace(" ￭","￭")
+        tokenized=tokenized.replace(" "," ▁")
+        tokenized=tokenized.replace("￭"," ")
+        tokenized=' '.join(tokenized.split()) 
+        return(tokenized)
+
+    def detokenize_sn(self,segment):
+        segment=self.detokenize_s(segment)
+        return(segment)        
+    
+def print_help():
+    print("MTUOC_tokenizer_cat.py A tokenizer for Spanish, usage:")
+    print("Simple tokenization:")
+    print('    cat "sentence to tokenize." | python3 MTUOC_tokenizer_spa.py tokenize')
+    print('    python3 MTUOC_tokenizer_spa.py tokenize < file_to_tokenize > tokenized_file')
+    print()
+    print("Simple detokenization:")
+    print('    cat "sentence to tokenize." | python3 MTUOC_tokenizer_spa.py detokenize')
+    print('    python3 MTUOC_tokenizer_spa.py detokenize < file_to_detokenize > detokenized_file')
+    print()
+    print("Advanced options:")
+    print("    tokenization/detokenization with joiner marks (￭): tokenize_j / detokenize_j")
+    print("    tokenization/detokenization with joiner marks (￭) and number splitting: tokenize_jn / detokenize_jn")
+    print("    tokenization/detokenization with splitter marks (▁): tokenize_s / detokenize_s")
+    print("    tokenization/detokenization with splitter marks (▁) and number splitting: tokenize_sn / detokenize_sn")
+        
+
+if __name__ == "__main__":
+    if len(sys.argv)>1:
+        if sys.argv[1]=="-h" or sys.argv[1]=="--help":
+            print_help()
+            sys.exit()
+        action=sys.argv[1]
+    else:
+        action="tokenize"
+    tokenizer=Tokenizer()
+    for line in sys.stdin:
+        line=line.strip()
+        #normalization of apostrophe
+        line=line.replace("’","'")
+        line=html.unescape(line)
+        if action=="tokenize":
+            outsegment=tokenizer.tokenize(line)
+        elif action=="detokenize":
+            outsegment=tokenizer.detokenize(line)
+        
+        elif action=="tokenize_s":
+            outsegment=tokenizer.tokenize_s(line)
+        elif action=="detokenize_s":
+            outsegment=tokenizer.detokenize_s(line)
+        elif action=="tokenize_sn":
+            outsegment=tokenizer.tokenize_sn(line)
+        elif action=="detokenize_sn":
+            outsegment=tokenizer.detokenize_sn(line)
+        
+        elif action=="tokenize_j":
+            outsegment=tokenizer.tokenize_j(line)
+        elif action=="detokenize_j":
+            outsegment=tokenizer.detokenize_j(line)
+        elif action=="tokenize_jn":
+            outsegment=tokenizer.tokenize_jn(line)
+        elif action=="detokenize_jn":
+            outsegment=tokenizer.detokenize_jn(line)
+        
+        print(outsegment)
+        

MTUOC_translate.py

@@ -0,0 +1,436 @@
+import config
+import sys
+import re
+import string as stringmodule
+
+
+import srx_segmenter
+
+from MTUOC_misc import printLOG
+from MTUOC_misc import capitalizeMTUOC
+if config.MTUOCServer_MTengine=="GoogleTranslate":
+    from MTUOC_GoogleTranslate import Google_translate
+if config.MTUOCServer_MTengine=="DeepL":
+    from MTUOC_DeepL import DeepL_translate
+if config.MTUOCServer_MTengine=="Lucy":
+    from MTUOC_Lucy import Lucy_translate
+from MTUOC_Marian import translate_segment_Marian
+from MTUOC_Moses import translate_segment_Moses
+
+from MTUOC_preprocess import preprocess_segment
+from MTUOC_preprocess import postprocess_segment
+
+from MTUOC_preprocess import tokenizationSL
+from MTUOC_preprocess import tokenizationTL
+from MTUOC_preprocess import detokenizationSL
+from MTUOC_preprocess import detokenizationTL
+
+def segmenta(cadena):
+    segmenter = srx_segmenter.SrxSegmenter(config.rules[config.SRXlang],cadena)
+    segments=segmenter.extract()
+    resposta=[]
+    return(segments)
+    
+def is_first_letter_upper(segment):
+    for character in segment:
+        if character.isalpha() and character.isupper():
+            return(True)
+        elif character.isalpha() and character.islower():
+            return(False)
+    return(False)
+
+def upper_case_first_letter(segment):
+    pos=0
+    for character in segment:
+        if character.isalpha() and character.islower():
+            llista=list(segment)
+            llista[pos]=llista[pos].upper()
+            segment="".join(llista)
+            return(segment)
+        elif character.isalpha() and character.isupper():
+            return(segment)
+        pos+=1
+    return(segment)
+    
+###URLs EMAILs
+
+def findEMAILs(string):
+    email=re.findall('\S+@\S+', string)
+    email2=[]
+    for em in email: 
+        if em[-1] in stringmodule.punctuation: em=em[0:-1]
+        email2.append(em)
+    return email2
+    
+def findURLs(string): 
+    regex = r"(?i)\b((?:https?://|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:'\".,<>?«»“”‘’]))"
+    url = re.findall(regex,string)       
+    return [x[0] for x in url] 
+
+def replace_EMAILs(string,code="@EMAIL@"):
+    EMAILs=findEMAILs(string)   
+    for EMAIL in EMAILs:
+        string=string.replace(EMAIL,code)
+    return(string)
+
+def replace_URLs(string,code="@URL@"):
+    URLs=findURLs(string)
+    for URL in URLs:
+        string=string.replace(URL,code)
+    return(string)
+    
+re_num = re.compile(r'[\d,\.]+')
+
+def replace_NUMs(segment,code="@NUM@"):
+    trobatsEXPRNUM=re.finditer(re_num,segment)
+    for trobat in trobatsEXPRNUM:
+        if not trobat.group(0) in [".",","]:
+            segment=segment.replace(trobat.group(0),code,1)
+    return(segment)
+
+def splitnumbers(segment,joiner=""):
+    joiner=joiner+" "
+    xifres = re.findall(re_num,segment)
+    equil={}
+    for xifra in xifres:
+        xifrastr=str(xifra)
+        xifrasplit=xifra.split()
+        xifra2=joiner.join(xifra)
+        segment=segment.replace(xifra,xifra2)
+        if xifra2.find(" ")>-1:
+            equil[xifra2]=xifra
+    return(segment,equil)
+    
+def desplitnumbers(segment,equil):
+    for xifra2 in equil:
+        segment=segment.replace(xifra2,equil[xifra2])
+    return(segment)
+   
+def restore_EMAILs(stringA,stringB,code="@EMAIL@"):
+    EMAILs=findEMAILs(stringA)
+    for email in EMAILs:
+        stringB=stringB.replace(code,email,1)
+    return(stringB)
+    
+def restore_URLs(stringA,stringB,code="@URL@"):
+    URLs=findURLs(stringA)
+    for url in URLs:
+        stringB=stringB.replace(code,url,1)
+    return(stringB)
+    
+def restore_NUMs(segmentSL,segmentTL,code="@NUM@"):
+    trobatsEXPRNUM=re.finditer(re_num,segmentSL)
+    position=0
+    for trobat in trobatsEXPRNUM:
+        if not trobat.group(0) in [".",","]:
+            segmentTL=segmentTL.replace(code,trobat.group(0),1)
+    return(segmentTL)
+ 
+
+def translate_para(paragraph):
+    if config.segment_input:
+        (segments,separators)=segmenta(paragraph)
+        translations=[]
+        for segment in segments:
+            translation=translate_segment(segment)
+            if config.fix_xml:
+                translation=config.tagrestorer.fix_xml_tags(translation)
+            translations.append(translation)
+        resultat=[]
+        for i in range(0,len(separators)):
+            resultat.append(separators[i])
+            try:
+                resultat.append(translations[i])
+            except:
+                pass
+        
+        
+            
+        translation="".join(resultat)
+        
+    else:
+        translation=translate_segment(paragraph)
+        
+    return(translation)
+        
+        
+
+
+def restore_tags_translation_candidates(translation_candidates):
+    hastags=config.tagrestorer.has_tags(translation_candidates["segmentTAGS"])
+    if hastags:
+        
+        (translation_candidates["segmentTAGS"],equil)=config.tagrestorer.replace_tags(translation_candidates["segmentOrig"])
+        printLOG(3,"replace_tags",translation_candidates["segmentTAGS"])
+        printLOG(3,"equil",equil)
+        (translation_candidates["segmentTAGS"],tagInici,tagFinal)=config.tagrestorer.remove_start_end_tag(translation_candidates["segmentOrig"])
+        printLOG(3,"remove_start_end_tag",translation_candidates["segmentTAGS"])
+        printLOG(3,"TAG initial:",tagInici)
+        printLOG(3,"TAG final:",tagFinal)
+        translation_candidates["segmentNOTAGS"]=config.tagrestorer.remove_tags(translation_candidates["segmentTAGS"])
+        originaltags=config.tagrestorer.get_tags(translation_candidates["segmentTAGS"])
+        segmentNOTAGSTOK=tokenizationSL(translation_candidates["segmentNOTAGS"])
+        segmentTAGSTOK=tokenizationSL(translation_candidates["segmentPreTAGS"])
+        translation_candidates["translationTAGS"]=[None] * len(translation_candidates["translationNOTAGSPre"])
+        for i in range(0,len(translation_candidates["translationNOTAGSPre"])):
+            try:
+                if hastags and config.tag_restoration:
+                    try:
+                        alignment=translation_candidates["alignments"][i]
+                        translationNOTAGSTOK=tokenizationTL(translation_candidates["translationNOTAGSPre"][i])
+                        translation_candidates["translationTAGS"][i]=config.tagrestorer.restore_tags(segmentNOTAGSTOK, segmentTAGSTOK, alignment, translationNOTAGSTOK)
+                        '''
+                        if tagInici:
+                            translation_candidates["translationTAGS"][i]=tagInici+translation_candidates["translationTAGS"][i]
+                        if tagFinal:
+                            translation_candidates["translationTAGS"][i]=translation_candidates["translationTAGS"][i]+tagFinal
+                        printLOG(3,"SELECTED TRANSLATION SIMPLE TAGS",translation_candidates["translationTAGS"][i])
+                        for t in equil:
+                            translation_candidates["translationTAGS"][i]=translation_candidates["translationTAGS"][i].replace(t,equil[t],1)
+                        '''
+                    except:
+                        printLOG(3,"ERROR restoring tags:",sys.exc_info())
+                        translation_candidates["translationTAGS"][i]=translationNOTAGSTOK
+                
+                else:
+                    translation_candidates["translationTAGS"]=translation_candidates["translationNOTAGSPre"]
+                    printLOG(3,"translationTAGS:",translation_candidates["translationTAGS"][i])
+                    
+                    
+                
+            except:
+                pass
+    else:
+        translation_candidates["segmentNOTAGS"]=translation_candidates["segmentTAGS"]
+        translation_candidates["translationTAGS"]=translation_candidates["translationNOTAGSPre"]
+
+    
+    
+    
+    return(translation_candidates)
+
+    
+def translate_segment(segment):
+    printLOG(3,"translate_segment",segment)
+    if config.MTUOCServer_MTengine=="GoogleTranslate":
+        translation=Google_translate(segment)
+        return(translation)
+    elif config.MTUOCServer_MTengine=="DeepL":
+        translation=DeepL_translate(segment)
+        return(translation)
+    elif config.MTUOCServer_MTengine=="Lucy":
+        translation=Lucy_translate(segment)
+        return(translation)
+    segmentOrig=segment
+    if not config.change_input_files[0]=="None":
+        printLOG(3,"CHANGES INPUT:")
+        printLOG(3,"ORIGINAL:",segmentOrig)
+        for change in config.changes_input:
+            tofind=change[0]
+            tochange=change[1]
+            regexp="\\b"+tofind+"\\b"
+            trobat=re.findall(regexp,segment)
+            if trobat:    
+                segment=re.sub(regexp, tochange, segment)
+                printLOG(3,tofind,tochange)
+        printLOG(3,"CHANGED:",segment)
+    hastags=config.tagrestorer.has_tags(segment)
+    originaltags=config.tagrestorer.get_tags(segment)
+    printLOG(3,"hastags",hastags)
+    printLOG(3,"originaltags",originaltags)
+    #truecasing
+    totruecase=False
+    toupperfinal=False
+    if not config.truecase==None and config.truecase=="all": totruecase=True
+    segmentnotags=config.tagrestorer.remove_tags(segment)
+    if not config.truecase==None and config.truecase in ["upper","all"] and segmentnotags.isupper() and not segment=="@URL@" and not segment=="@EMAIL@": 
+        totruecase=True
+        toupperfinal=True
+    if config.checkistranslatable:
+        segmentNOTAGS=replace_URLs(segment,config.code_URLs)
+        segmentNOTAGS=replace_EMAILs(segment,config.code_EMAILs)
+        tokens=tokenizationSL(segmentNOTAGS)
+        if not is_translatable(tokens): 
+            return(segment)
+    if totruecase:        
+        segment=config.truecaser.truecase(segment)
+    if config.tokenizerSL:
+        segment=config.tokenizerSL.tokenize(segment)
+    if hastags:
+        segmentTAGS=segment
+        
+        (segmentTAGS,equil)=config.tagrestorer.replace_tags(segmentTAGS)
+        printLOG(3,"segmentTAGS:",segmentTAGS)
+        printLOG(3,"equil:",equil)
+        printLOG(3,"segmentTAGS:",segmentTAGS)
+        (segmentTAGS,tagInici,tagFinal)=config.tagrestorer.remove_start_end_tag(segmentTAGS)
+        printLOG(3,"TAG initial:",tagInici)
+        printLOG(3,"TAG final:",tagFinal)
+        
+        
+        segmentNOTAGS=config.tagrestorer.remove_tags(segment)
+    else:
+        segmentTAGS=segment
+        segmentNOTAGS=segment
+    if len(segmentNOTAGS)<config.min_chars_segment:
+        return(segment)
+        
+    
+    
+    if config.MTUOCServer_EMAILs:
+        segmentTAGS=replace_EMAILs(segmentTAGS)
+        segmentNOTAGS=replace_EMAILs(segmentNOTAGS)
+        printLOG(3,"Replace EMAILs:",segmentTAGS)
+    if config.MTUOCServer_URLs:
+        segmentTAGS=replace_URLs(segmentTAGS)
+        segmentNOTAGS=replace_URLs(segmentNOTAGS)
+        printLOG(3,"Replace URLs:",segmentTAGS)
+        
+    if config.pre_replace_NUMs:
+        segmentTAGS=replace_NUMs(segmentTAGS,code=config.code_NUMs)
+        segmentNOTAGS=replace_NUMs(segmentNOTAGS,code=config.code_NUMs)
+        printLOG(3,"Replace NUMs:",segmentTAGS)
+    if config.pre_split_NUMs:
+        (segmentTAGS,equilSplitNum)=splitnumbers(segmentTAGS) 
+        (segmentNOTAGS,equilSplitNum2)=splitnumbers(segmentNOTAGS)  
+        printLOG(3,"Split NUMs:",segmentTAGS)        
+    
+    
+    #leading and trailing spaces
+    leading_spaces=len(segment)-len(segment.lstrip())
+    trailing_spaces=len(segment)-len(segment.rstrip())-1
+    segmentPre=preprocess_segment(segmentNOTAGS)  
+    segmentPreTAGS=preprocess_segment(segmentTAGS)     
+    if config.MTUOCServer_MTengine=="Marian":
+        translation_candidates=translate_segment_Marian(segmentPre)
+    elif config.MTUOCServer_MTengine=="Moses":
+        translation_candidates=translate_segment_Moses(segmentPre)
+        
+    translation_candidates["segment"]=segment
+    translation_candidates["segmentOrig"]=segmentOrig
+    translation_candidates["segmentTAGS"]=segmentTAGS
+    translation_candidates["segmentPre"]=segmentPre
+    translation_candidates["segmentPreTAGS"]=segmentPreTAGS
+    translation_candidates["segmentNOTAGS"]=segmentNOTAGS
+    print("*****1",translation_candidates)
+    if hastags:
+        translation_candidates=restore_tags_translation_candidates(translation_candidates)  
+        #(translation_candidates["segmentTAGS"],equil)=config.tagrestorer.replace_tags(translation_candidates["segmentOrig"])
+     
+    else:
+        translation_candidates["translationTAGS"]=translation_candidates["translationNOTAGSPre"]
+    print("*****2",translation_candidates)      
+    for i in range(0,len(translation_candidates["translationNOTAGSPre"])):
+        translation_candidates["translationTAGS"][i]=postprocess_segment(translation_candidates["translationTAGS"][i])
+        
+        if hastags:
+            if tagInici:
+                translation_candidates["translationTAGS"][i]=tagInici+translation_candidates["translationTAGS"][i]
+            if tagFinal:
+                translation_candidates["translationTAGS"][i]=translation_candidates["translationTAGS"][i]+tagFinal
+            for t in equil:
+                translation_candidates["translationTAGS"][i]=translation_candidates["translationTAGS"][i].replace(t,equil[t],1)
+            #if not config.truecaser==None and is_first_letter_upper(segmentOrig):    
+            #    translation_candidates["translationTAGS"][i]=upper_case_first_letter(translation_candidates["translationTAGS"][i])
+        print("*****3",translation_candidates)     
+        if totruecase:
+            translation_candidates["translationTAGS"][i]=capitalizeMTUOC(translation_candidates["translationTAGS"][i])
+        if toupperfinal: 
+            translation_candidates["translationTAGS"][i]=translation_candidates["translationTAGS"][i].upper()
+            ###LOWERCASE UPPERCASED TAGS
+            hastagstranslation=config.tagrestorer.has_tags(translation_candidates["translationTAGS"][i])
+            if hastagstranslation:
+                translationtags=config.tagrestorer.get_tags(translation_candidates["translationTAGS"][i])
+                for tt in translationtags:
+                    if not tt in originaltags and tt.lower() in originaltags:
+                        translation_candidates["translationTAGS"][i]=translation_candidates["translationTAGS"][i].replace(tt,tt.lower())
+        
+        if config.MTUOCServer_EMAILs:
+            translation_candidates["translationTAGS"][i]=restore_EMAILs(segmentOrig,translation_candidates["translationTAGS"][i],code=config.code_EMAILs)
+        if config.MTUOCServer_URLs:
+            translation_candidates["translationTAGS"][i]=restore_URLs(segmentOrig,translation_candidates["translationTAGS"][i],code=config.code_URLs)
+        '''    
+        #config.pre_replace_NUMs
+        if config.pre_replace_NUMs:
+            translation_candidates["translationTAGS"][i]=restore_NUMs(segmentOrig,translation_candidates["translationTAGS"][i],code=config.code_NUMs)
+        #config.pre_split_NUMs
+        if config.pre_split_NUMs:
+            translation_candidates["translationTAGS"][i]=desplitnumbers(translation_candidates["translationTAGS"][i],equilSplitNum)
+        #detruecase
+        if totruecase:
+            translation_candidates["translationTAGS"][i]=translation_candidates["translationTAGS"][i][0].upper()+translation_candidates["translationTAGS"][i][1:]
+        #detokenize
+        '''
+        if config.tokenizerSL and not config.tokenizerTL==None:
+            translation_candidates["translationTAGS"][i]=config.tokenizerTL.detokenize(translation_candidates["translationTAGS"][i])
+            
+        
+                
+        translation_candidates["translationTAGS"][i]=config.tagrestorer.repairSpacesTags(translation_candidates["segmentOrig"],translation_candidates["translationTAGS"][i]) 
+        printLOG(3,"SELECTED TRANSLATION REAL TAGS",translation_candidates["translationTAGS"][i])
+    best_translation=select_best_candidate(translation_candidates,config.translation_selection_strategy)
+    
+    translation=best_translation
+    
+    if not config.change_output_files[0]=="None":
+        printLOG(3,"CHANGES OUTPUT:")
+        printLOG(3,"ORIGINAL:",translation)
+        for change in config.changes_output:
+            tofind=change[0]
+            tochange=change[1]
+            regexp="\\b"+tofind+"\\b"
+            trobat=re.findall(regexp,translation)
+            if trobat: 
+                translation=re.sub(regexp, tochange, translation)
+                printLOG(3,tofind,tochange)
+        printLOG(3,"CHANGED:",translation)
+    
+    if not config.change_translation_files[0]=="None":
+        printLOG(3,"CHANGES TRANSLATION:")
+        printLOG(3,"ORIGINAL SOURCE:",segmentOrig)
+        printLOG(3,"ORIGINAL TARGET:",translation)
+        for change in config.changes_translation:
+            tofindSOURCE=change[0]
+            tofindTARGET=change[1]
+            tochange=change[2]
+            regexpSOURCE="\\b"+tofindSOURCE+"\\b"
+            regexpTARGET="\\b"+tofindTARGET+"\\b"
+            trobatSOURCE=re.findall(regexpSOURCE,segmentOrig)
+            trobatTARGET=re.findall(regexpTARGET,translation)
+            if trobatSOURCE and trobatTARGET: 
+                translation=re.sub(regexpTARGET, tochange, translation)
+                printLOG(3,tofindTARGET,tochange)
+        printLOG(3,"CHANGED TARGET:",translation)
+    
+    return(translation)
+
+def is_translatable_old(tokens):
+    tokens=tokens.split(" ")
+    translatable=False
+    for token in tokens:
+        if token.isalpha():
+            translatable=True
+            break
+    return(translatable)
+
+def is_translatable(tokens):    
+    translatable=False
+    for token in tokens.split():
+        transtoken=True
+        for character in token:
+            if str(character) in ["0","1","2","3","4","5","6","7","8","9"]:
+                transtoken=False
+                break
+        if transtoken:
+            translatable=True
+    return(translatable) 
+
+def select_best_candidate(translation_candidates,strategy):
+    '''To implement several strategies to select the best candidate. Now it r,eturns the first one.'''
+    if strategy=="First":
+        best_translation=translation_candidates["translationTAGS"][0]
+    return(best_translation)
+
+    
+
+  

MTUOC_truecaser.py

@@ -0,0 +1,199 @@
+#    MTUOC_truecaser
+#    v. 07/06/2023
+#    Copyright (C) 2021  Antoni Oliver
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+import sys
+import codecs
+import pickle
+import argparse
+import importlib
+
+class Truecaser():
+    def __init__(self, MTUOCPath=".", tokenizer=None, tc_model=None):
+        
+        self.initchars=["¡","¿","-","*","+","'",'"',"«","»","—","‘","’","“","”","„",]
+        
+        if tokenizer==None:
+            self.tokenizer=None
+        else:
+            sys.path.append(MTUOCPath)
+            if tokenizer.endswith(".py"):tokenizer=tokenizer.replace(".py","")
+            self.module = importlib.import_module(tokenizer)
+            self.tokenizer=self.module.Tokenizer()
+        if tc_model:
+            self.tc_model = pickle.load(open(tc_model, "rb" ) )
+        else:
+            self.tc_model={}
+    def set_tc_model(self, tc_model):
+        self.tc_model = pickle.load(open(tc_model, "rb" ) )
+        
+    def set_tokenizer(self, tokenizer):
+        if tokenizer.endswith(".py"):tokenizer=tokenizer.replace(".py","")
+        self.module = importlib.import_module(tokenizer)
+        self.tokenizer=self.module.Tokenizer()
+    
+    def set_MTUOCPath(self, path):
+        sys.path.append(path)
+        
+    def isinitsymbol(self, token):
+        if len(token)==1 and token in self.initchars:
+            return(True)
+        else:
+            return(False)
+        
+    def detect_type(self, segment):
+        tipus="unknown"
+        if self.tokenizer==None:
+            tokens=segment.split(" ")
+        else:
+            tokens=self.tokenizer.tokenize(segment)
+        ntok=0
+        utokens=0
+        ltokens=0
+        leadingsplitter=False
+        trailingsplitter=False
+        leadingjoiner=False
+        trailingsplitter=False
+        for token in tokens:
+            token=token.replace("▁","")
+            token=token.replace("￭","")
+            if token.isalpha():ntok+=1
+            if token.isalpha() and token==token.lower():
+                ltokens+=1
+            elif token.isalpha() and not token==token.lower():
+                utokens+=1
+        if ntok>=5 and utokens>=ntok/2 and not segment==segment.upper():
+            tipus="titled"
+        elif segment==segment.upper():
+            tipus="uppercased"
+        else:
+            tipus="regular"
+        return(tipus)
+        
+    def truecase(self, line, ucf=False, restoreCase=False):
+        if self.tokenizer:
+            tokens=self.tokenizer.tokenize_s(line).split(" ")
+        else:
+            tokens=line.split(" ")
+        nsegment=[]
+        for token in tokens:
+            
+            try:
+                leadingsplitter=False
+                trailingsplitter=False
+                leadingjoiner=False
+                trailingjoiner=False
+                if token.startswith("▁"):leadingsplitter=True
+                if token.endswith("▁"):trailingsplitter=True
+                if token.startswith("￭"):leadingjoiner=True
+                if token.endswith("￭"):trailingjoiner=True
+                token=token.replace("▁","")
+                token=token.replace("￭","")
+                try:
+                    nlc=self.tc_model[token.lower()]["lc"]
+                except:
+                    nlc=0
+                try:
+                    nu1=self.tc_model[token.lower()]["u1"]
+                except:
+                    nu1=0
+                try:
+                    nuc=self.tc_model[token.lower()]["uc"]
+                except:
+                    nuc=0
+                proceed=False
+                if not token==token.lower(): proceed=True
+                if restoreCase: proceed=True
+                if proceed:
+                    if nlc>0 and nlc>=nu1 and nlc>=nuc:
+                        token=token.lower()
+                    elif nu1>0 and nu1>nlc and nu1>nuc:
+                        token=token.lower().capitalize()
+                    elif nuc>0 and nuc>nlc and nuc>nu1:
+                        token=token.upper()
+
+                if leadingsplitter:token="▁"+token
+                if trailingsplitter:token=token+"▁"
+                if leadingjoiner:token="￭"+token
+                if trailingjoiner:token=token+"￭"
+                nsegment.append(token)
+            except:
+                print("ERROR",sys.exc_info())
+                nsegment.append(token)
+        if self.tokenizer:
+            nsegment=self.tokenizer.detokenize_s(" ".join(nsegment))     
+        else:
+            nsegment=" ".join(nsegment)
+        if ucf:
+            if self.isinitsymbol(nsegment[0]):firstchar=1
+            else: firstchar=0
+            try:
+                if firstchar==0:
+                    nsegment=nsegment[firstchar].upper()+"".join(nsegment[1:])
+                elif firstchar==1:
+                    nsegment=nsegment[0]+nsegment[firstchar].upper()+"".join(nsegment[2:])
+            except:
+                pass
+        return(nsegment)
+        
+    def detruecase_old(self,line,tokenizer):
+        tokens=line.split(" ")
+        new=[]
+        yet=False
+        if tokenizer:
+            tokens=tokenizer.tokenize_j(line).split(" ")
+        else:
+            tokens=line.split(" ")
+        for token in tokens:
+            if not yet and token.isalpha():
+                yet=True
+                new.append(token[0].upper()+token[1:])
+            else:
+                new.append(token)
+        line=" ".join(new)
+        detrue=tokenizer.detokenize_j(line)
+        return(line) 
+
+    def detruecase(self,line):
+        detruecased=line.capitalize()
+        return(line) 
+    
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description='MTUOC program for truecasing.')
+    parser.add_argument('-m','--model', action="store", dest="model", help='The truecasing model to use.',required=True)
+    parser.add_argument('-t','--tokenizer', action="store", dest="tokenizer", help='The tokenizer to used',required=False)
+    parser.add_argument('-u','--ucf', action="store_true", dest="ucf", help='Set if you want first word capitalized',required=False)
+    parser.add_argument('-r','--restore', action="store_true", dest="restore", help='Set if you want to restore case (uppercase lower cased)',required=False)
+    parser.add_argument('--mtuoc','--MTUOC', action="store", dest="MTUOC", help='The path to the MTUOC components',required=False)
+    
+    args = parser.parse_args()
+    model=args.model
+    ucf=args.ucf
+    restore=args.restore
+        
+
+    if args.MTUOC:
+        MTUOCPath=args.MTUOC
+    else:
+        MTUOCPath=""
+
+    truecaser=Truecaser(MTUOCPath, args.tokenizer, args.model)
+    for line in sys.stdin:
+        line=line.strip()
+        tcline=truecaser.truecase(line, ucf, restore)
+        print(tcline)
+    

MTUOC_typeMTUOC.py

@@ -0,0 +1,75 @@
+#    MTUOC_tupeMTUOC
+#    Copyright (C) 2023  Antoni Oliver
+#    v. 07/06/2023
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+import sys
+from flask import Flask, jsonify, request, make_response
+
+from MTUOC_misc import get_IP_info
+from MTUOC_misc import printLOG
+import config
+
+from MTUOC_translate import translate_para
+
+def start_MTUOC_server():
+    cli = sys.modules['flask.cli']
+    cli.show_server_banner = lambda *x: None
+    STATUS_OK = "ok"
+    STATUS_ERROR = "error"
+    printLOG(1,config.verbosity_level,"MTUOC server started using MTUOC protocol")
+    STATUS_ERROR = "error"
+    out={}
+    def start(url_root="./translator",
+              host="0.0.0.0",
+              port=5000,
+              debug=True):
+        def prefix_route(route_function, prefix='', mask='{0}{1}'):
+            def newroute(route, *args, **kwargs):
+                return route_function(mask.format(prefix, route), *args, **kwargs)
+            return newroute
+
+        app = Flask(__name__)
+        app.route = prefix_route(app.route, url_root)
+
+        @app.route('/translate', methods=['POST'])
+        def translateMTUOC():
+            try:
+                body = request.get_json()
+                ts=translate_para(body["src"])
+                jsonObject = {
+                    "id": body["id"],
+                    "src": body["src"],
+                    
+                    "tgt": ts # Call MTUOC
+                }
+                printLOG(0,config.verbosity_level,jsonObject)
+                return jsonify(jsonObject)
+            except:
+                return make_response("Server Error", 500)
+        
+        from waitress import serve
+        serve(app, host=host, port=port,threads= 1)    
+        #app.run(debug=debug, host=host, port=port, use_reloader=False,threaded=True)
+    url_root="/"
+    ip="0.0.0.0"
+    ip=get_IP_info()
+    debug="store_true"
+    
+    print("MTUOC server IP:   ", ip)
+    print("MTUOC server port: ", config.MTUOCServer_port)
+    print("MTUOC server type:  MTUOC")
+    
+    start(url_root=url_root, host=ip, port=config.MTUOCServer_port,debug=debug)

MTUOC_typeModernMT.py

@@ -0,0 +1,69 @@
+#    MTUOC_typeModernMT
+#    Copyright (C) 2023  Antoni Oliver
+#    v. 07/06/2023
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+import sys
+from flask import Flask, jsonify, request
+
+from MTUOC_misc import get_IP_info
+from MTUOC_misc import printLOG
+import config
+
+from MTUOC_translate import translate_para
+
+def start_ModernMT_server():
+    cli = sys.modules['flask.cli']
+    cli.show_server_banner = lambda *x: None
+    STATUS_OK = "ok"
+    STATUS_ERROR = "error"
+    printLOG(1,"MTUOC server started as ModernMT server")
+    def start(
+              url_root="",
+              host="",
+              port=config.MTUOCServer_port,
+              debug=True):
+        def prefix_route(route_function, prefix='', mask='{0}{1}'):
+            def newroute(route, *args, **kwargs):
+                return route_function(mask.format(prefix, route), *args, **kwargs)
+            return newroute
+        app = Flask(__name__)
+        app.route = prefix_route(app.route, url_root)
+        @app.route('/translate', methods=['GET'])
+        def translateModernMT():
+            out = {}
+            try:
+                out['data']={}
+                segment=request.args['q']
+                translation=translate_para(segment)
+                out['data']['translation']=translation
+            except:
+                out['status'] = STATUS_ERROR
+            return jsonify(out)
+        #ip=get_IP_info()
+        #app.run(debug=True, host=ip, port=MTUOCServer_port, use_reloader=False, threaded=True)
+    #start()
+        from waitress import serve
+        serve(app, host=host, port=port,threads= 8)    
+        #app.run(debug=debug, host=host, port=port, use_reloader=False,threaded=True)
+    url_root="/"
+    ip="0.0.0.0"
+    ip=get_IP_info()
+    debug="store_true"
+    
+    print("MTUOC server IP:   ", ip)
+    print("MTUOC server port: ", config.MTUOCServer_port)
+    print("MTUOC server type:  ModernMT")
+    
+    start(url_root=url_root, host=ip, port=config.MTUOCServer_port,debug=debug)

MTUOC_typeMoses.py

@@ -0,0 +1,45 @@
+#    MTUOC_typeMoses
+#    Copyright (C) 2023  Antoni Oliver
+#    v. 07/06/2023
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+import sys
+from xmlrpc.server import SimpleXMLRPCServer
+
+from MTUOC_misc import get_IP_info
+from MTUOC_misc import printLOG
+import config
+
+from MTUOC_translate import translate_para
+
+def translate(segment):
+    #function for Moses server
+    translation=translate_para(segment['text'])
+    translationdict={}
+    translationdict["text"]=translation
+    return(translationdict) 
+
+def start_Moses_server():
+    server = SimpleXMLRPCServer(("", config.MTUOCServer_port),logRequests=True,)
+    server.register_function(translate)
+    server.register_introspection_functions()
+    # Start the server
+    try:
+        ip=get_IP_info()
+        printLOG(1,"MTUOC server IP:   ", ip)
+        printLOG(1,"MTUOC server port: ", config.MTUOCServer_port)
+        printLOG(1,"MTUOC server type:  Moses")
+        server.serve_forever()
+    except KeyboardInterrupt:
+        printLOG(1,'Exiting')

MTUOC_typeNMTWizard.py

@@ -0,0 +1,68 @@
+#    MTUOC_typeNMTWizard
+#    Copyright (C) 2023  Antoni Oliver
+#    v. 07/06/2023
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+import sys
+from flask import Flask, jsonify, request
+
+from MTUOC_misc import get_IP_info
+from MTUOC_misc import printLOG
+import config
+
+from MTUOC_translate import translate_para
+
+def start_NMTWizard_server():
+    cli = sys.modules['flask.cli']
+    cli.show_server_banner = lambda *x: None
+    STATUS_OK = "ok"
+    STATUS_ERROR = "error"
+    printLOG(1,"MTUOC server started as NMTWizard server")
+    out={}
+    def start(url_root="",
+              host="0.0.0.0",
+              port=5000,
+              debug=True):
+        def prefix_route(route_function, prefix='', mask='{0}{1}'):
+            def newroute(route, *args, **kwargs):
+                return route_function(mask.format(prefix, route), *args, **kwargs)
+            return newroute
+
+        app = Flask(__name__)
+        app.route = prefix_route(app.route, url_root)
+
+        @app.route('/translate', methods=['POST'])
+        def translateONMT():
+            inputs = request.get_json(force=True)
+            sourcetext=inputs["src"][0]["text"]
+            try:
+                targettext=translate_para(sourcetext)
+                out={"tgt": [[{"text": targettext}]]}
+            except:
+                out['error'] = "Error"
+                out['status'] = STATUS_ERROR
+            return jsonify(out)
+        from waitress import serve
+        serve(app, host=host, port=port,threads= 1)    
+        #app.run(debug=debug, host=host, port=port, use_reloader=False,threaded=True)
+    url_root="/"
+    ip="0.0.0.0"
+    ip=get_IP_info()
+    debug="store_true"
+    
+    print("MTUOC server IP:   ", ip)
+    print("MTUOC server port: ", config.MTUOCServer_port)
+    print("MTUOC server type:  NMTWizard")
+    
+    start(url_root=url_root, host=ip, port=config.MTUOCServer_port,debug=debug)

MTUOC_typeOpenNMT.py

@@ -0,0 +1,78 @@
+#    MTUOC_typtNMT
+#    Copyright (C) 2023  Antoni Oliver
+#    v. 07/06/2023
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+import sys
+from flask import Flask, jsonify, request
+
+from MTUOC_misc import get_IP_info
+from MTUOC_misc import printLOG
+import config
+
+from MTUOC_translate import translate_para
+
+def start_OpenNMT_server():
+    cli = sys.modules['flask.cli']
+    cli.show_server_banner = lambda *x: None
+    STATUS_OK = "ok"
+    STATUS_ERROR = "error"
+    printLOG(1,"MTUOC server started as OpenNMT server")
+    STATUS_ERROR = "error"
+    out={}
+    def start(url_root="./translator",
+              host="0.0.0.0",
+              port=5000,
+              debug=True):
+        def prefix_route(route_function, prefix='', mask='{0}{1}'):
+            def newroute(route, *args, **kwargs):
+                return route_function(mask.format(prefix, route), *args, **kwargs)
+            return newroute
+
+        app = Flask(__name__)
+        app.route = prefix_route(app.route, url_root)
+
+        @app.route('/translate', methods=['POST'])
+        def translateONMT():
+            inputs = request.get_json(force=True)
+            inputs0=inputs[0]
+            out = {}
+            try:
+                out = [[]]
+                ss=inputs0['src']
+                ts=translate_para(ss)
+                response = {"src": ss, "tgt": ts,
+                                "n_best": 0, "pred_score": 0}
+                    
+                out[0].append(response)
+            except:
+                out['error'] = "Error"
+                out['status'] = STATUS_ERROR
+
+            return jsonify(out)
+            
+        
+        from waitress import serve
+        serve(app, host=host, port=port,threads= 1)    
+        #app.run(debug=debug, host=host, port=port, use_reloader=False,threaded=True)
+    url_root=config.MTUOCServer_ONMT_url_root
+    ip="0.0.0.0"
+    ip=get_IP_info()
+    debug="store_true"
+    
+    print("MTUOC server IP:   ", ip)
+    print("MTUOC server port: ", config.MTUOCServer_port)
+    print("MTUOC server type:  OpenNMT")
+    
+    start(url_root=url_root, host=ip, port=config.MTUOCServer_port,debug=debug)

README.md

@@ -1,3 +1 @@
----
-license: gpl-3.0
----
+# MTUOC-server

config-server.yaml

@@ -0,0 +1,115 @@
+MTEngine:
+    MTengine: Marian
+    #one of Marian, OpenNMT, Moses, GoogleTranslate, DeepL, Lucy
+    startMTEngine: True
+    startCommand: "./marian-server-GPU -m model.bin -v vocab-es.yml vocab-en.yml -p 8200 --n-best  --alignment hard  --normalize 1 --quiet &"
+    IP: localhost
+    port: 8200
+    min_len_factor: 0.5
+    
+MTUOCServer:
+  port: 8000
+  type: MTUOC
+  #one of MTUOC, Moses, ModernMT, OpenNMT, NMTWizard
+  verbosity_level: 3
+  
+  checkistranslatable: True
+  restore_tags: True
+  fix_xml: True
+  
+  URLs: True
+  code_URLs: "@URL@"
+  EMAILs: True
+  code_EMAILs: "@EMAIL@"
+  
+  replaceNUMs: True
+  
+  splitNUMs: False
+  
+  min_chars_segment: 2  
+  translation_selection_strategy: First
+  log_file: log.log
+  ONMT_url_root: "/translator"
+  
+SRX_file: segment.srx
+
+
+Preprocess:
+  sl_lang: es
+  tl_lang: en
+  tokenize_SL: False
+  sl_tokenizer: MTUOC_tokenizer_spa.py
+  #one of the MTUOC-tokenizers or state Moses if you use Moses tokenizers
+  tl_tokenizer: MTUOC_tokenizer_eng.py
+  #one of the MTUOC-tokenizers or state Moses if you use Moses tokenizers
+  
+  segment_input: True
+  SRXfile: segmentSC.srx
+  SRXlang: Spanish
+  
+  replaceNUMs: False
+  code_NUMs: "@NUM@"
+  splitNUMs: False
+  #Truecasing
+  tcmodel: tc.es
+  #or None if not used
+  truecase: upper
+  #one of None, all, upper
+  ###SentencePiece###
+  sentencepiece: True
+  sp_model_SL: spmodel.model
+  sp_model_TL: spmodel.model
+  sp_splitter: "▁"
+  
+  #BPE
+  BPE: False
+  #True or False: if sentencepiece is set to True, sentencepiece will be used
+  bpecodes: codes_file
+  bpe_joiner: "@@"
+  
+  bos_annotate: True
+  bos_symbol: <s>
+  #None or <s> (or other)
+  eos_annotate: True
+  eos_symbol: </s>
+
+#hmtl entities
+
+unescape_html_input: True
+escape_html_input: False
+
+unescape_html_output: True
+escape_html_output: False 
+
+change_input_files: None
+change_output_files: None
+change_translation_files: None
+  
+GoogleTranslate:
+  sllang: en
+  tllang: es
+  glossary: None
+  #state None if no glossary is used, otherwise the name of the glossary
+  project_id: XXxxXXXXX
+  location: us-central1
+  jsonfile: XXxxXXXXX.json
+  
+DeepL:
+  API_key: XXxxXXXXX
+  sllang: EN
+  tllang: ES
+  #tag_handling: html
+  #one of html, xml
+  formality: default
+  #one of default, less, more
+  split_sentences: "off"
+  #one of "off", "on", "nonewlines"
+  glossary: None
+  
+Lucy:
+  url: http://XXX.XXX.XXX:8080/AutoTranslateRS/V1.3/mtrans/exec/
+  TRANSLATION_DIRECTION: SPANISH-CATALAN
+  MARK_UNKNOWNS: 0
+  MARK_ALTERNATIVES: 0
+  MARK_COMPOUNDS: 0
+  CHARSET: UTF

config.py

@@ -0,0 +1,114 @@
+MTUOCServer_port=""
+MTUOCServer_MTengine=""
+startMTEngineV=False
+startMTEngineCommand=""
+MTUOCServer_type=""
+MTEngineIP=""
+MTEnginePort=""
+verbosity_level=0
+log_file=""
+sortidalog=""
+min_len_factor=0.5
+min_chars_segment=1
+tag_restoration=False
+
+
+segment_input=False
+SRXfile=None
+SRXlang=None
+rules=None
+
+sllang=""
+tllang=""
+tokenize_SL=False
+tokenize_TL=False
+tokenizerSL=None
+tokenizerTL=None
+tokenizerSLType=""
+tokenizerTLType=""
+
+tcmodel=""
+truecase=""
+truecaser=None
+checkistranslatable=False
+
+MTUOCServer_URLs=False
+MTUOCServer_EMAILs=False
+
+code_URLs="@URL@"
+code_EMAILs="@EMAIL@"
+
+pre_replace_NUMs=False
+code_NUMs="@NUM@"
+pre_split_NUMs=False
+
+#sentencepiece
+sentencepiece=""
+spmodelSL=""
+spmodelTL=""
+sp_splitter=""
+
+#BPE
+BPE=""
+bpecodes=""
+bpe_joiner=""
+bpeobject=None
+
+
+leading_spaces=0
+trailing_spaces=0
+
+MTUOCServer_NUMs=False
+MTUOCServer_splitNUMs=False
+
+escape_html_input=False
+unescape_html_input=True
+
+change_input_files=None
+change_output_files=None
+change_translation_files=None
+changes_input=[]
+changes_output=[]
+changes_translation=[]
+
+
+tagrestorer=None
+
+bos_annotate=False
+bos_symbol="<s>"
+#None or <s> (or other)
+eos_annotate=False
+eos_symbol="</s>"
+
+spSL=None
+spTL=None
+
+translation_selection_strategy="First"
+
+Google_sllang=""
+Google_tllang=""
+Google_glossary=""
+Google_project_id=""
+Google_location=""
+Google_jsonfile=""
+client=None
+parent=None
+
+DeepL_API_key=""
+DeepL_sl_lang=""
+DeepL_tl_lang=""
+DeepL_formality=""
+DeepL_split_sentences=""
+DeepL_glossary=""
+DeepLtranslator=None
+
+Lucy_url=""
+Lucy_TRANSLATION_DIRECTION=""
+Lucy_MARK_UNKNOWNS=""
+Lucy_MARK_ALTERNATIVES=""
+Lucy_MARK_COMPOUNDS=""
+Lucy_CHARSET=""
+
+MTUOCServer_ONMT_url_root=None
+
+proxyMoses=None

marian-server-GPU

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5146caf265138f2b7fd84cfa69309072e1c2128600afdd857a199593aff4d7b3
+size 487624648

model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:229f37d69935a792c5c2a01dc8bb961ca8364a6e3cbc57ee3d6e206dd27e5c69
+size 528817672

model.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:352051764d584a60946b7133f0e268e9c4af70e0bacbbf5ae00142caea1759b7
+size 528823268

requirements.txt

@@ -0,0 +1,27 @@
+pip3 install websockets
+pip3 install lxml
+pip3 install regex
+pip3 install websocket
+pip3 install websocket-client
+pip3 install requests
+pip3 install flask
+pip3 install PyYAML
+pip3 install sentencepiece
+pip3 install waitress
+
+Per GoogleTranslate
+pip3 install google-cloud
+pip3 install google-cloud-vision
+pip3 install google-cloud-translate
+ip3 install google-api-python-client
+
+
+Per DeepL
+pip3 install deepl
+
+Per Lucy
+pip3 install xmltodict
+
+
+Per poder crear executables
+pip3 install pyinstaller

spmodel.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c314df995019c6bbe81cc35452090270f3e481db8b2306f73e5d565ef774c260
+size 1349219

srx_segmenter.py

@@ -0,0 +1,106 @@
+"""Segment text with SRX.
+"""
+__version__ = '0.0.2'
+
+import lxml.etree
+import regex
+from typing import (
+    List,
+    Set,
+    Tuple,
+    Dict,
+    Optional
+)
+
+
+class SrxSegmenter:
+    """Handle segmentation with SRX regex format.
+    """
+    def __init__(self, rule: Dict[str, List[Tuple[str, Optional[str]]]], source_text: str) -> None:
+        self.source_text = source_text
+        self.non_breaks = rule.get('non_breaks', [])
+        self.breaks = rule.get('breaks', [])
+
+    def _get_break_points(self, regexes: List[Tuple[str, str]]) -> Set[int]:
+        return set([
+            match.span(1)[1]
+            for before, after in regexes
+            for match in regex.finditer('({})({})'.format(before, after), self.source_text)
+        ])
+
+    def get_non_break_points(self) -> Set[int]:
+        """Return segment non break points
+        """
+        return self._get_break_points(self.non_breaks)
+
+    def get_break_points(self) -> Set[int]:
+        """Return segment break points
+        """
+        return self._get_break_points(self.breaks)
+
+    def extract(self) -> Tuple[List[str], List[str]]:
+        """Return segments and whitespaces.
+        """
+        non_break_points = self.get_non_break_points()
+        candidate_break_points = self.get_break_points()
+
+        break_point = sorted(candidate_break_points - non_break_points)
+        source_text = self.source_text
+
+        segments = []  # type: List[str]
+        whitespaces = []  # type: List[str]
+        previous_foot = ""
+        for start, end in zip([0] + break_point, break_point + [len(source_text)]):
+            segment_with_space = source_text[start:end]
+            candidate_segment = segment_with_space.strip()
+            if not candidate_segment:
+                previous_foot += segment_with_space
+                continue
+
+            head, segment, foot = segment_with_space.partition(candidate_segment)
+
+            segments.append(segment)
+            whitespaces.append('{}{}'.format(previous_foot, head))
+            previous_foot = foot
+        whitespaces.append(previous_foot)
+
+        return segments, whitespaces
+
+
+def parse(srx_filepath: str) -> Dict[str, Dict[str, List[Tuple[str, Optional[str]]]]]:
+    """Parse SRX file and return it.
+    :param srx_filepath: is soruce SRX file.
+    :return: dict
+    """
+    tree = lxml.etree.parse(srx_filepath)
+    namespaces = {
+        'ns': 'http://www.lisa.org/srx20'
+    }
+
+    rules = {}
+
+    for languagerule in tree.xpath('//ns:languagerule', namespaces=namespaces):
+        rule_name = languagerule.attrib.get('languagerulename')
+        if rule_name is None:
+            continue
+
+        current_rule = {
+            'breaks': [],
+            'non_breaks': [],
+        }
+
+        for rule in languagerule.xpath('ns:rule', namespaces=namespaces):
+            is_break = rule.attrib.get('break', 'yes') == 'yes'
+            rule_holder = current_rule['breaks'] if is_break else current_rule['non_breaks']
+
+            beforebreak = rule.find('ns:beforebreak', namespaces=namespaces)
+            beforebreak_text = '' if beforebreak.text is None else beforebreak.text
+
+            afterbreak = rule.find('ns:afterbreak', namespaces=namespaces)
+            afterbreak_text = '' if afterbreak.text is None else afterbreak.text
+
+            rule_holder.append((beforebreak_text, afterbreak_text))
+
+        rules[rule_name] = current_rule
+
+    return rules

start

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b884a8ee6c060a3332d6f72e716ab72f6ec855668ac3080853ee2136cc9eecf9
+size 5742224

stop

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:59ce505c15d74857bbfa9095dbf5c0aa0bd601d4d251b074fbdb2956f21a97b9
+size 5628368

tc.es

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f2fa32066ef0df781509a2c3817073f56d6398f69a7121993b1f7431ac984b90
+size 147397351