Package: tika
tika-pythonA Python port of theApache Tikalibrary that makes Tika available using theTika REST Server.This makes Apache Tika available as a Python library,
installable via Setuptools, Pip and Easy Install.To use this library, you need to have Java 7+ installed on your
system as tika-python starts up the Tika REST server in the
background.Inspired byAptivate Tika.Installation (with pip)pip install tikaInstallation (without pip)python setup.py buildpython setup.py installAirgap Environment SetupTo get this working in a disconnected environment, download a tika server file (both tika-server.jar and tika-server.jar.md5, which can be foundhere) and set the TIKA_SERVER_JAR environment variable to TIKA_SERVER_JAR="file:////tika-server.jar" which successfully tellspython-tikato "download" this file and move it to/tmp/tika-server.jarand run as background process.This is the only way to runpython-tikawithout internet access. Without this set, the default is to check the tika version and pull latest every time from Apache.Environment VariablesThese are read once, when tika/tika.py is initially loaded and used throughout after that.TIKA_VERSION- set to the version string, e.g., 1.12 or default to current Tika version.TIKA_SERVER_JAR- set to the full URL to the remote Tika server jar to download and cache.TIKA_SERVER_ENDPOINT- set to the host (local or remote) for the running Tika server jar.TIKA_CLIENT_ONLY- if set to True, thenTIKA_SERVER_JARis ignored, and relies on the value forTIKA_SERVER_ENDPOINTand treats Tika like a REST client.TIKA_TRANSLATOR- set to the fully qualified class name (defaults to Lingo24) for the Tika translator implementation.TIKA_SERVER_CLASSPATH- set to a string (delimited by ':' for each additional path) to prepend to the Tika server jar path.TIKA_LOG_PATH- set to a directory with write permissions and thetika.logandtika-server.logfiles will be placed in this directory.TIKA_PATH- set to a directory with write permissions and thetika_server.jarfile will be placed in this directory.TIKA_JAVA- set the Java runtime name, e.g.,javaorjava9TIKA_STARTUP_SLEEP- number of seconds (float) to wait per check if Tika server is launched at runtimeTIKA_STARTUP_MAX_RETRY- number of checks (int) to attempt for Tika server startup if launched at runtimeTIKA_JAVA_ARGS- set java runtime arguments, e.g,-Xmx4gTIKA_LOG_FILE- set the filename for the log file. default:tika.log. if it is an empty string (''), no log file is created.Testing it outParser Interface (backwards compat prior to REST)#!/usr/bin/env pythonimporttikatika.initVM()fromtikaimportparserparsed=parser.from_file('/path/to/file')print(parsed["metadata"])print(parsed["content"])Parser InterfaceThe parser interface extracts text and metadata using the /rmeta
interface. This is one of the better ways to get the internal XHTML
content extracted.Note:The parser interface needs the following environment variable set on the console for printing of the extracted content.export PYTHONIOENCODING=utf8#!/usr/bin/env pythonimporttikafromtikaimportparserparsed=parser.from_file('/path/to/file')print(parsed["metadata"])print(parsed["content"])Optionally, you can pass Tika server URL along with the call
what's useful for multi-instance execution or when Tika is dockerzed/linked.parsed=parser.from_file('/path/to/file','http://tika:9998/tika')string_parsed=parser.from_buffer('Good evening, Dave','http://tika:9998/tika')You can also pass a binary streamwith open(file, 'rb') as file_obj:
    response = tika.parser.from_file(file_obj)Gzip compressionSince Tika 1.24.1 gzip compression of input and output streams is allowed.Input compression can be achieved with gzip or zlib:import zlib 

    with open(file, 'rb') as file_obj:
        return tika.parser.from_buffer(zlib.compress(file_obj.read()))

...

    import gzip
 
    with open(file, 'rb') as file_obj:
        return tika.parser.from_buffer(gzip.compress(file_obj.read()))And output with the header:with open(file, 'rb') as file_obj:
        return tika.parser.from_file(file_obj, headers={'Accept-Encoding': 'gzip, deflate'})Specify Output Format To XHTMLThe parser interface is optionally able to output the content as XHTML rather than plain text.Note:The parser interface needs the following environment variable set on the console for printing of the extracted content.export PYTHONIOENCODING=utf8#!/usr/bin/env pythonimporttikafromtikaimportparserparsed=parser.from_file('/path/to/file',xmlContent=True)print(parsed["metadata"])print(parsed["content"])# Note: This is also available when parsing from the buffer.Unpack InterfaceThe unpack interface handles both metadata and text extraction in a single
call and internally returns back a tarball of metadata and text entries that
is internally unpacked, reducing the wire load for extraction.#!/usr/bin/env pythonimporttikafromtikaimportunpackparsed=unpack.from_file('/path/to/file')Detect InterfaceThe detect interface provides a IANA MIME type classification for the
provided file.#!/usr/bin/env pythonimporttikafromtikaimportdetectorprint(detector.from_file('/path/to/file'))Config InterfaceThe config interface allows you to inspect the Tika Server environment's
configuration including what parsers, mime types, and detectors the
server has been configured with.#!/usr/bin/env pythonimporttikafromtikaimportconfigprint(config.getParsers())print(config.getMimeTypes())print(config.getDetectors())Language Detection InterfaceThe language detection interface provides a 2 character language
code texted based on the text in provided file.#!/usr/bin/env pythonfromtikaimportlanguageprint(language.from_file('/path/to/file'))Translate InterfaceThe translate interface translates the text automatically extracted
by Tika from the source language to the destination language.#!/usr/bin/env pythonfromtikaimporttranslateprint(translate.from_file('/path/to/spanish','es','en'))Using a BufferNote you can also use a Parser and Detector
.from_buffer(string|BufferedIOBase) method to dynamically parser
a string or bytes buffer in Python and/or detect its MIME
type. This is useful if you've already loaded
the content into memory.string_parsed=parser.from_buffer('Good evening, Dave')byte_data:bytes=b'B\xc3\xa4ume'parsed=parser.from_buffer(io.BytesIO(byte_data))Using Client Only ModeYou can set Tika to use Client only mode by settingimporttikatika.TikaClientOnly=TrueThen you can run any of the methods and it will fully
omit the check to see if the service on localhost is
running and omit printing the check messages.Changing the Tika ClasspathYou can update the classpath that Tika server uses by
setting the classpath as a set of ':' delimited strings.
For example if you want to get Tika-Python working withGeoTopicParsing,
you can do this, replace paths below with your own paths, as
identifiedhereand make sure that you have done this:kill Tika server (if already running):psaux|grepjava|grepTikakill-9PIDimporttika.tikaimportosfromtikaimportparserhome=os.getenv('HOME')tika.tika.TikaServerClasspath=home+'/git/geotopicparser-utils/mime:'+home+'/git/geotopicparser-utils/models/polar'parsed=parser.from_file(home+'/git/geotopicparser-utils/geotopics/polar.geot')printparsed["metadata"]Customizing the Tika Server RequestYou may customize the outgoing HTTP request to Tika server by settingrequestOptionson the.from_fileand.from_buffermethods (Parser, Unpack , Detect, Config, Language, Translate). It should be a dictionary of arguments that will be passed to the request method. Therequest method documentationspecifies valid arguments. This will override any defaults except forurlandparams/data.fromtikaimportparserparsed=parser.from_file('/path/to/file',requestOptions={'timeout':120})New Command Line Client ToolWhen you install Tika-Python you also get a new command
line client tool,tika-pythoninstalled in your /path/to/python/bin
directory.The options and help for the command line tool can be seen by typingtika-pythonwithout any arguments. This will also download a copy of
the tika-server jar and start it if you haven't done so already.tika.py[-v][-o<outputDir>][--server<TikaServerEndpoint>][--install<UrlToTikaServerJar>][--port<portNumber>]<command><option><urlOrPathToFile>

tika.pyparsealltest.pdftest2.pdf(writeoutputJSONmetadatafilesfortest1.pdf_meta.jsonandtest2.pdf_meta.json)tika.pydetecttypetest.pdf(returnsmime-typeastext/plain)tika.pylanguagefilefrench.txt(returnslanguagee.g.,frastext/plain)tika.pytranslatefr:enfrench.txt(translatesthefilefrench.txtfromfrenchtoenglish)tika.pyconfigmime-types(seewhatmime-typestheTikaServercanhandle)Asimplepythonandcommand-lineclientforTikausingthestandaloneTikaserver(JARfile).
AllcommandsreturnresultsinJSONformatbydefault(excepttextintext/plain).

Toparsedocs,use:
tika.pyparse<meta|text|all><path>

TochecktheconfigurationoftheTikaserver,use:
tika.pyconfig<mime-types|detectors|parsers>

Commands:parse=parsetheinputfileandwriteaJSONdocfile.ext_meta.jsoncontainingtheextractedmetadata,text,orbothdetecttype=parsethestreamand'detect'theMIME/mediatype,returnintext/plainlanguagefile=parsethefilestreamandidentifythelanguageofthetext,returnits2charactercodeintext/plaintranslatesrc:dest=parseandextracttextandthentranslatethetextfromsourcelanguagetodestinationlanguageconfig=returnaJSONdocdescribingtheconfigurationoftheTikaserver(i.e.mime-typesitcanhandle,orinstalleddetectorsorparsers)Arguments:urlOrPathToFile=filetobeparsed,ifURLitwillfirstberetrievedandthenpassedtoTika

Switches:--verbose,-v=verbosemode--encode,-e=encoderesponseinUTF-8--csv,-c=reportdetectoutputincomma-delimitedformat--server<TikaServerEndpoint>=usearemoteTikaServeratthisendpoint,otherwiseuselocalserver--install<UrlToTikaServerJar>=downloadandexecTikaServer(JARfile),startingserverondefaultport9998Exampleusageaspythonclient:
--fromtikaimportrunCommand,parse1
--jsonOutput=runCommand('parse','all',filename)or
--jsonOutput=parse1('all',filename)Questions, comments?Send them toChris A. Mattmann.ContributorsChris A. Mattmann, JPLBrian D. Wilson, JPLDongni Zhao, USCKenneth Durri, University of MarylandTyler Palsulich, New York University & GoogleJoe Germuska, Northwestern UniversityVlad Shvedov, Profinda.comDiogo Vieira, Globo.comAron Ahmadia, Continuum AnalyticsKaranjeet Singh, USCRenat Nasyrov, YandexJames Brooking, BlackbeardYash Tanna, USCIgor Tokarev, FreelanceImraan Parker, FreelanceAnnie K. Didier, JPLJuan Elosua, TEGRA Cybersecurity CenterCarina de Oliveira Antunes, CERNThanksThanks to theDARPA MEMEXprogram for funding most of the original portions of this work.LicenseApache License, version 2