Package: act-scio
act-scio2Scio v2 is a reimplementation ofScioin Python3.Scio usestikato extract text from documents (PDF, HTML, DOC, etc).The result is sent to the Scio Analyzer that extracts information using a combination of NLP
(Natural Language Processing) and pattern matching.Changelog0.0.42SCIO now supports setting TLP on data upload, to annotate documents withtlptag. Documents downloaded by feeds will have a default TLP white, but this can be changed in the config for feeds.Source codeThe source code the workers are available ongithub.SetupTo setup, first install from PyPi:sudopip3installact-scioYou will also need to installbeanstalkd. On debian/ubuntu you can run:sudoaptinstallbeanstalkdConfigure beanstalk to accept larger payloads with the-zoption. For red hat derived setups this can be configured in/etc/sysconfig/beanstalkd:MAX_JOB_SIZE=-z524288You then need to install NLTK data files. A helper utility to do this is included:scio-nltk-downloadYou will also need to create a default configuration:scio-configuserAPITo run the api, execute:scio-apiThis will setup the API on 127.0.0.1:3000. Use--port <PORT> and --host <IP>to listen on another port and/or another interface.For documentation of the API endpoint seeAPI.md.ConfigurationYou can create a default configuration using this command (should be run as the user running scio):scio-configuserCommon configuration can be found under ~/.config/scio/etc/scio.iniRunning ManuallyScio Tika ServerThe Scio Tika server reads jobs from the beanstalk tubescio_docand the extracted text will be sent to the tubescio_analyze.The first time the server runs, it will download tika using maven. It will use a proxy if$https_proxyis set.scio-tika-serverscio-tika-serverusestika-pythonwhich depends on tika-server.jar. If your server has internet access, this will downloaded automatically. If not or you need proxy to connect to the internet, follow the instructions on "Airagap Environment Setup" here:https://github.com/chrismattmann/tika-python. Currently only tested with tika-server version 2.7.0.Scio Analyze ServerScio Analyze Server reads (by default) jobs from the beanstalk tubescio_analyze.scio-analyzeYou can also read directly from stdin like this:echo"The companies in the Bus; Finanical, Aviation and Automobile industry are large."|scio-analyze--beanstalk=--elasticsearch=Scio SubmitSubmit document (from file or URI) toscio_api.Example:scio-submit\--urihttps://www2.fireeye.com/rs/848-DID-242/images/rpt-apt29-hammertoss.pdf\--scio-baseurihttp://localhost:3000/submit\--tlpwhiteRunning as a serviceSystemd compatible service scripts can be found under examples/systemd.To install:sudocpexamples/systemd/*.service/usr/lib/systemd/system
sudosystemctlenablescio-tika-server
sudosystemctlenablescio-analyze
sudoservicestartscio-tika-server
sudoservicestartscio-analyzescio-feed cron jobTo continously fetch new content from feeds, you can add scio-feed to cron like this (make sure the directory $HOME/logs exists):# Fetch scio feeds every hour
0 * * * * /usr/local/bin/scio-feeds >> $HOME/logs/scio-feed.log.$(date +\%s) 2>&1

# Delete logs from scio-feeds older than 7 days
0 * * * * find $HOME/logs/ -name 'scio-feed.log.*' -mmin +10080 -exec rm {} \;Local developmentUse pip to install inlocal development mode. act-scio uses namespacing, so it is not compatible with usingsetup.py installorsetup.py develop.In repository, run:pip3install--user-e.