Package: opentrep
Open Travel Request Parser (TREP)Table of contentOpen Travel Request Parser (TREP)Table of contentOverviewDocker imagesOpenTREP Docker imagesGeneral purpose C++/Python Docker imagesCentOS 7Ubuntu 18.04 LTS Bionic BeaverDebian 9 StretchCommon to all the above-mentioned Linux distributionsNative installation (without Docker)RPM-based distributions (eg, Fedora/CentOS/RedHat)Installation from the sourcesClone the Git repositoryAlternatively, download and extract the tar-ballInstallation of the dependenciesOn LinuxFedoraCentOSDebian/UbuntuOn MacOSICUBoostCentOSSOCIGeneral Unix/LinuxDebianMacOSBuilding the library and test binaryUnderlying (relational) database, SQLite or MySQL/MariaDB, if anyIndexing the POR dataFilling the (relational) database, SQLite or MySQL/MariaDB, if anyXapian indexing with standard installationXapian indexing for an ad hoc deployed Web applicationSearchingDeployment stagesIndex, or not, non-IATA PORInstalling a Python virtual environmentChecking that the Python module worksTrouble-shooting Python issues on MacOSInterceptors not installed / lateOpenTREP as a Python extensionReferencesBuild and package OpenTREP as a Python extensionInstall Python modules/dependenciesInstall OpenTrep Python extension with system-basedpipBuild the OpenTrep Python extension locally with system-based Scikit-buildTest the OpenTREP Python extensionUse the OpenTREP Python extensionDownload the latest OpenTravelData (OPTD) POR data fileXapian index initializationSearch with the OpenTrep Python extensionSearch with the OpenTrepWrapper package(Optional) Running the Django-based application server (needs update)Table of contents generated with markdown-tocOverviewOpenTREPaims at providing a clean API, and the corresponding C++
implementation, for parsing travel-/transport-focused requests.
It powers thehttps://transport-search.orgWeb site (as well
as its newer version,https://www2.transport-search.org).OpenTREP uses Xapian (https://www.xapian.org) for the Information Retrieval
part, on freely available transport-/travel-related data (e.g., country
names and codes, city names and codes, airline names and codes,etc),
mainly to be found in theOpenTravelData (OPTD)
project:http://github.com/opentraveldata/opentraveldata/tree/master/opentraveldataOpenTREP exposes a simple, clean and object-oriented, API. For instance,
theOPENTREP::interpretTravelRequest()method:takes, as input, a character string containing the travel request;and yields, as output, the list of the recognized terms,
as well as their corresponding types.As an example, the travel requestWashington DC Beijing Monday a/r +AA -UA 1 week 2 adults 1 dogwould yield
the following list:Origin airport: Washington, DC, United States (US)Destination airport: Beijing, China (CN)Date of travel: next MondayDate of return: 1 week after next MondayPreferred airline: American Airlines (AA);
non-preferred airline: United AirlinesNumber of travelers: 2 adults and a dogThe output may then be used by other systems, for instance to book the
corresponding travel, or to visualize it on a map and calendar and to
share it with others.Note that the current version of OpenTREP recognizes only geographical POR
(points of reference), whatever their number in the request. For instance,
the requestlviv rivne jitomir kbp kharkiv dnk ods lwoyields the
following list of POR:LWO,RWN,ZTR,KBP,HRK,DNK,ODSandLWOagain.
Seethat request in action on the transport-search.org siteorthrough the API(enableJSONViewor similar for a more confortable reading).OpenTREP also deals with transport-related requests. For instance,cnshg deham nlrtm uslbg brssz cnshgcorrespond to a world tour
of famous ports:Shanghai, China (CN)Hamburg, Germany (DE)Rotterdam, Netherlands (NL)Los Angeles/Long Beach, California (CA), United States (US)Santos, Brazil (BR)(Back to) Shanghai, China (CN)
Seethat request in action on the transport-search.org siteorthrough the API.The underlying data for the POR is theOpenTravelDataoptd_por_public_all.csvfile.
A good complementary tool isGeoBase,
a Python-based software able to access any travel-related data source.OpenTREP makes an extensive use of existing open-source libraries for
increased functionality, speed and accuracy. In particular theBoost(C++ Standard Extensions),XapianandSOCIlibraries are used.Docker imagesOpenTREP Docker imagesDocker images provide ready-to-use environments, and are available onDocker CloudandQuay.io:$dockerpullopentrep/search-travel:legacy# for Docker.io$dockerpullquay.io/trep/opentrep# for Quay.io$dockerrun--rm-itopentrep/search-travel:legacybashSeehttps://github.com/trep/opentrep/tree/master/gui/legacyfor more details.General purpose C++/Python Docker imagesGeneral purpose Docker images for C++/Python development are also available fromDocker Cloud.
Those Docker images allow to develop on the major Linux distributions,i.e.,
CentOS, Debian and Ubuntu.CentOS 8$dockerpullcpppythondevelopment/base:centos8
$dockerrun-tcpppythondevelopment/base:centos8bash[build@2..c~]$$mkdir-p~/dev/geo&&cd~/dev/geo[build@2..cgeo]$gitclonehttps://github.com/trep/opentrep.git[build@2..cgeo]$cdopentrep&&mkdirbuild&&cdbuild[build@2..cbuild(master)]$cmake3-DCMAKE_INSTALL_PREFIX=${HOME}/dev/deliveries/opentrep-99.99.99-DCMAKE_BUILD_TYPE:STRING=Debug-DINSTALL_DOC:BOOL=ON-DRUN_GCOV:BOOL=OFF-DLIB_SUFFIX=..CentOS 7$dockerpullcpppythondevelopment/base:centos7
$dockerrun-tcpppythondevelopment/base:centos7bash[build@2..c~]$$mkdir-p~/dev/geo&&cd~/dev/geo[build@2..cgeo]$gitclonehttps://github.com/trep/opentrep.git[build@2..cgeo]$cdopentrep&&mkdirbuild&&cdbuild[build@2..cbuild(master)]$cmake3-DCMAKE_INSTALL_PREFIX=${HOME}/dev/deliveries/opentrep-99.99.99-DCMAKE_BUILD_TYPE:STRING=Debug-DINSTALL_DOC:BOOL=ON-DRUN_GCOV:BOOL=OFF-DLIB_SUFFIX=..Ubuntu 18.04 LTS Focal Fossal$dockerpullcpppythondevelopment/base:ubuntu2004
$dockerrun-tcpppythondevelopment/base:ubuntu2004bash[build@2..c~]$$mkdir-p~/dev/geo&&cd~/dev/geo[build@2..cgeo]$gitclonehttps://github.com/trep/opentrep.git[build@2..cgeo]$cdopentrep&&mkdirbuild&&cdbuild[build@2..cbuild(master)]$cmake-DCMAKE_INSTALL_PREFIX=${HOME}/dev/deliveries/opentrep-99.99.99-DCMAKE_BUILD_TYPE:STRING=Debug-DINSTALL_DOC:BOOL=ON-DRUN_GCOV:BOOL=OFF-DLIB_SUFFIX=..Ubuntu 18.04 LTS Bionic Beaver$dockerpullcpppythondevelopment/base:ubuntu1804
$dockerrun-tcpppythondevelopment/base:ubuntu1804bash[build@2..c~]$$mkdir-p~/dev/geo&&cd~/dev/geo[build@2..cgeo]$gitclonehttps://github.com/trep/opentrep.git[build@2..cgeo]$cdopentrep&&mkdirbuild&&cdbuild[build@2..cbuild(master)]$cmake-DCMAKE_INSTALL_PREFIX=${HOME}/dev/deliveries/opentrep-99.99.99-DCMAKE_BUILD_TYPE:STRING=Debug-DINSTALL_DOC:BOOL=ON-DRUN_GCOV:BOOL=OFF-DLIB_SUFFIX=..Debian 10 Buster$dockerpullcpppythondevelopment/base:debian10
$dockerrun-tcpppythondevelopment/base:debian10bash[build@2..c~]$$mkdir-p~/dev/geo&&cd~/dev/geo[build@2..cgeo]$gitclonehttps://github.com/trep/opentrep.git[build@2..cgeo]$cdopentrep&&mkdirbuild&&cdbuild[build@2..cbuild(master)]$cmake-DCMAKE_INSTALL_PREFIX=${HOME}/dev/deliveries/opentrep-99.99.99-DCMAKE_BUILD_TYPE:STRING=Debug-DINSTALL_DOC:BOOL=ON-DRUN_GCOV:BOOL=OFF-DLIB_SUFFIX=..Debian 9 Stretch$dockerpullcpppythondevelopment/base:debian9
$dockerrun-tcpppythondevelopment/base:debian9bash[build@2..c~]$$mkdir-p~/dev/geo&&cd~/dev/geo[build@2..cgeo]$gitclonehttps://github.com/trep/opentrep.git[build@2..cgeo]$cdopentrep&&mkdirbuild&&cdbuild[build@2..cbuild(master)]$cmake-DCMAKE_INSTALL_PREFIX=${HOME}/dev/deliveries/opentrep-99.99.99-DCMAKE_BUILD_TYPE:STRING=Debug-DINSTALL_DOC:BOOL=ON-DRUN_GCOV:BOOL=OFF-DLIB_SUFFIX=..Common to all the above-mentioned Linux distributions[build@2..cbuild(master)]$makeinstall[build@2..cbuild(master)]$./opentrep/opentrep-indexer[build@2..cbuild(master)]$./opentrep/opentrep-searcher-q"nice san francisco"Native installation (without Docker)RPM-based distributions (eg, Fedora/CentOS/RedHat)Since OpenTREP has been approved as an official package of Fedora/CentOS/RedHat
(see thereview request on Bugzilla for further details),
just useDNF(orYumfor the older distributions):$dnf-yinstallopentrepopentrep-docInstallation from the sourcesConfigure the environmentIf not already specified, the following environment variables may be set:exportplatform="$(uname|tr'[:upper:]''[:lower:]')"exportarchitecture="$(uname-m|sed's/x86_/amd/')"Clone the Git repositoryThe GitHub repository may be cloned as following:$mkdir-p~/dev/geo&&cd~/dev/geo
$gitclonehttps://github.com/trep/opentrep.git
$cdopentrep
$gitcheckoutmasterAlternatively, download and extract the tar-ballGitHub generatestar-balls on the fly for every tagged release.
For instance:$wgethttps://github.com/trep/opentrep/archive/opentrep-0.07.10.tar.gzNote that SourceForge also stores someolder archived tar-balls.Installation of the dependenciesOn LinuxThe following packages may be needed (Fedora/RedHat/CentOS names on the left
hand side, Debian/Ubuntu names on the right hand side; names for other
Linux distributions may vary):cmake(orcmake3on CentOS)gcc-c++/gcc,g++boost-devel(orboost169-develon CentOS) /libboost-all-devxapian-core-devel/libxapian-devpython-devel/python,libpython-devlibicu-devel/libicu-devsoci-mysql-devel,soci-sqlite3-devel/ SOCI needs to be installed from
the sources on Debian, Ubuntu and MacOS (see the dedicated section below)sqlite3-devel/sqlite3,libsqlite3-devmariadb-devel/libmariadb-dev,libmysql++-devprotobuf-devel,protobuf-compiler/libprotobuf-dev,protobuf-compilerreadline-devel/libreadline-dev,libncurses5-devghostscriptdoxygen(optional)tetex-latex/texlive-latex-recommended(optional)rpm-buildFor instance, the following subsections show respective installation commands
for a few famous Linux distributions.FedoraGeneral C++-base stack:$dnf-yinstallgit-allbash-completiongcc-c++cmakeboost-devel\xapian-core-develsoci-mysql-develsoci-sqlite3-develreadline-devel\sqlite-develmariadb-devellibicu-develprotobuf-develprotobuf-compiler\autoconfautomakelibtoolPython development (the package names may vary; Python 2 is no longer
supported):$dnf-yinstallpython-develpython-pipRelated Python packages (the package names may vary):$dnf-yinstallpython3-protobufboost-python3-develDocumentation tools:$dnf-yinstalldoxygenghostscript"tex(latex)"CentOSIf DNF is not already installed (e.g., on CentOS 7):$yum-yinstalldnfGeneral C++-based stack:$dnf-yinstallgit-allbash-completiongcc-c++cmakeboost169-devel\xapian-core-develsoci-mysql-develsoci-sqlite3-develreadline-devel\sqlite-develmariadb-devellibicu-develprotobuf-develprotobuf-compiler\autoconfautomakelibtoolRelated Python packages (the package names may vary):$dnf-yinstallpython3-protobufboost169-python3-develPython development (the package names may vary):$dnf-yinstallpython39python39-devel
$dnf-yinstallpython3python3-develDocumentation tools:$dnf-yinstalldoxygenghostscript"tex(latex)"Debian/UbuntuVarious C++ and Python packages:$apt-get-yinstalllocales&&locale-gen"en_US.UTF-8"$apt-get-yinstallzlib1g-devlibbz2-devlsb-releaselibgmp-dev\libgmp-devgccg++clangcppcheckcmakelibboost-all-devlibxapian-dev\graphvizlibreadline-devlibncurses5-devlibczmq-devlibzmq3-dev\libssl-devlibffi-devsqlite3libsqlite3-devlibmariadb-dev\libmysql++-devpostgresql-server-dev-alllibicu-dev\libprotobuf-devprotobuf-compilerPython development (the package names may vary)$apt-get-yinstallpython3libpython3-devpython3-pipDocumentation tools$apt-get-yinstalldoxygenghostscripttexlive-latex-recommendedOn MacOSWithHomebrew:$brewinstallboostboost-python3cmakelibedit\xapiansqlitemysqlmysql-clienticu4cprotobufprotobuf-cdoxygen
$brewinstallhomebrew/portable-ruby/portable-readlineNote that, as of end 2022, the Hombrew recipes for Python 3 are now
specific up to the minor version, more specifically:Python 3.11:python@3.11(Python 3.11.0 as of November 2022)Python 3.10:python@3.10(Python 3.10.8 as of November 2022)Python 3.9:python@3.9(Python 3.9.15 as of November 2022)Python 3.8:python@3.8(Python 3.8.15 as of November 2022)Previously, the default Python 3 installation was Python 3.8 (now part of
thepython@3.8Homebrew package).
As a reminder, on MacOS with Homebrew, a way to get the details is:$brewinfopython@3.11
python@3.11:stable3.11.0(bottled)$brewinfopython@3.10
python@3.10:stable3.10.8(bottled)$brewinfopython@3.9
python@3.9:stable3.9.15(bottled)$brewinfopython@3.8
python@3.8:stable3.8.15(bottled)Because of the multiple parallel installations of Python versions,
the following setup may be necessary:$sudomkdir-p/usr/local/Cellar/python
$sudochown-R$USER/usr/local/Cellar/python
$ln-s/usr/local/Cellar/python@3.11/3.11.0/usr/local/Cellar/python/3.11.0
$ln-s/usr/local/Cellar/python@3.10/3.10.8/usr/local/Cellar/python/3.10.8
$ln-s/usr/local/Cellar/python@3.9/3.9.15/usr/local/Cellar/python/3.9.15
$ln-s/usr/local/Cellar/python@3.8/3.8.15/usr/local/Cellar/python/3.8.15Clean links on potential older versions:$ls-lFh/usr/local/Cellar/python/
total0lrwxr-xr-x1userstaff34BJan921:143.8.11@->/usr/local/Cellar/python@3.8/3.8.11
lrwxr-xr-x1userstaff34BJan921:143.9.6@->/usr/local/Cellar/python@3.9/3.9.6
$sudounlink/usr/local/Cellar/python/3.8.7Check the following links, as installed by Homebrew on MacOS:$ls-lFh/usr/local/Frameworks/Python.framework/Versions/Current
lrwxr-xr-x1userstaff79BOct1312:01/usr/local/Frameworks/Python.framework/Versions/Current@->../../../Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/Current
$ls-lFh/usr/local/Frameworks/Python.framework/Versions/3.10
lrwxr-xr-x1userstaff76BOct1312:00/usr/local/Frameworks/Python.framework/Versions/3.10@->../../../Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10
$ls-lFh/usr/local/Frameworks/Python.framework/Versions/3.11
lrwxr-xr-x1userstaff76BNov3017:37/usr/local/Frameworks/Python.framework/Versions/3.11@->../../../Cellar/python@3.11/3.11.0/Frameworks/Python.framework/Versions/3.11
$ls-lFh/usr/local/Frameworks/Python.framework/Versions/3.9
lrwxr-xr-x1darnaudstaff75BOct1312:01/usr/local/Frameworks/Python.framework/Versions/3.9@->../../../Cellar/python@3.9/3.9.15/Frameworks/Python.framework/Versions/3.9If those links are not as expected (as of January 2022, they were correct),
recreate them:$sudochown-R$USER/usr/local/Frameworks/Python.framework
$unlink/usr/local/Frameworks/Python.framework/Versions/3.11
$ln-s/usr/local/Cellar/python@3.11/3.11.0/Frameworks/Python.framework/Versions/3.11/usr/local/Frameworks/Python.framework/Versions/3.11
$unlink/usr/local/Frameworks/Python.framework/Versions/Current
$ln-s/usr/local/Cellar/python@3.11/3.11.0/Frameworks/Python.framework/Versions/3.11/usr/local/Frameworks/Python.framework/Versions/CurrentLeading to:$ls-lFh/usr/local/Cellar/python/
total0lrwxr-xr-x1useradmin34BJan921:143.10.1@->/usr/local/Cellar/python@3.10/3.10.1
lrwxr-xr-x1useradmin34BJan921:143.9.9@->/usr/local/Cellar/python@3.9/3.9.9
lrwxr-xr-x1useradmin34BJan921:143.8.12@->/usr/local/Cellar/python@3.8/3.8.12
$ls-lFh/usr/local/Frameworks/Python.framework/Versions/
total0lrwxr-xr-x1userstaff75BJan904:013.9@->../../../Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9
lrwxr-xr-x1userstaff79BJan904:01Current@->../../../Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/CurrentUp until recently (mid-2020), Boost.Python came with a dependency on
thelibpythonPython library. As of July 2020, it seems to no longer
be the case. As a matter of fact, as stated in theWhat's new documentation of Python 3.8:On Unix, C extensions are no longer linked to libpython except on Android
and Cygwin. It is now possible for a statically linked Python to load
a C extension built using a shared library Python.
(Contributed by Victor Stinner inbpo-21536.).The rationale is that most of the C extensions (including OpenTREP
Python extension and Boost.Python) are meant to be loaded by the Python
interpreter, which brings its own Python library (libpython).
So, if the C extensions are linked againstlibpythonthemselves,
it is highly probable that the versions of both thoselibpythonlibraries (the one integrated with the Python interpreter and
the one linked with the C extension) would differ.The adoption of that change (no longer linking a C extension withlibpython) is progressing slowly, and highly dependent on the
C extensions and the platforms. For instance:On Fedora Linux distributions, that change was implemented only
from Boost 1.73 on Fedora 33 (released at the end of 2020).On MacOS, that change seems to have been implemented as a patch to
Boost 1.72 package; as it can be seen below, Boost 1.72+
on MacOS no longer seems to be linked againstlibpython.$brewinfoboost-python3
boost-python3:stable1.76.0(bottled),HEAD
/usr/local/Cellar/boost-python3/1.76.0(472files,18.2MB)*
$ls-lFh/usr/local/Cellar/boost-python3/1.76.0/lib/libboost_python39.dylib
-r--r--r--1userstaff410KMay2621:13/usr/local/Cellar/boost-python3/1.76.0/lib/libboost_python39.dylib
$otool-L/usr/local/Cellar/boost-python3/1.76.0/lib/libboost_python39.dylib
/usr/local/Cellar/boost-python3/1.76.0/lib/libboost_python39.dylib:/usr/local/opt/boost-python3/lib/libboost_python39.dylib(compatibilityversion0.0.0,currentversion0.0.0)/usr/lib/libc++.1.dylib(compatibilityversion1.0.0,currentversion905.6.0)/usr/lib/libSystem.B.dylib(compatibilityversion1.0.0,currentversion1292.100.5)The OpenTREP Python extension itself has adopted that change with
version 0.07.7 release at the end of May 2020.On some platforms, it may still happen that Boost.Python
and the local Python are based on different versions. That may lead
to some weird crashes not obvious to debug.ICUInstall ICU with Homebrew$brewinstallicu4cBoostFollow the instructions onBoost helper documentation on GitHubto install Python and Boost on some platforms, including MacOS.CentOSOn CentOS, the version of Boost is often outdated, for instanceBoost 1.53 on CentOS 7.
It may have adversarial effects, for instance on CentOS 7, wherePython 3 libraries have been upgraded from Python 3.4 to
Python 3.6,
but where Boost 1.53 Python libraries have not been upgraded accordingly,
leading for the least tolinking errors.Hopefully,EPELprovides
more recent Boost releases, for instanceBoost 1.69 on RHEL/CentOS 7.Those additional specific Boost packages may be installed in parallel
to the official Boost packages.
CMake has however to be made aware of that additional Boost package and,
equally importantly, of its location on the file-system.Example of additional CMake configuration variables forBoost 1.69 on CentOS 7:$cmake3[...]\-DBOOST_LIBRARYDIR=/usr/lib64/boost169\-DBOOST_INCLUDEDIR=/usr/include/boost169\-DBoost_ADDITIONAL_VERSIONS="1.69 1.69.0"\[...]SOCIOn CentOS and Fedora, since the author of this package (OpenTREP)
is also the official maintainer of the SOCI package, that latter
is usually up-to-dateOn Debian, Ubuntu and MacOS however, SOCI must be built from the sources.
The following shows how to do that on MacOS (on Debian/Ubuntu, one can have
a look at the part installing SOCI on theC++/Python Docker files)Download and prepare SOCI:$sudomkdir-p/opt/soci&&sudochown-R${USER}/opt/soci
$gitclonehttps://github.com/SOCI/soci.git/opt/soci/socigitGeneral Unix/Linux$mkdir-p/opt/soci/socigit/build/head
$pushd/opt/soci/socigit/build/head
$cmake-DCMAKE_INSTALL_PREFIX=/usr-DCMAKE_BUILD_TYPE=Release-DSOCI_CXX11=ON-DSOCI_TESTS=OFF../..
$make
$sudomakeinstall
$popdDebian$wgethttps://github.com/trep/opentrep/raw/master/ci-scripts/soci-debian-cmake.patch-O/opt/soci/soci-debian-cmake.patch
$pushd/opt/soci/socigit
$patch-p1<../soci-debian-cmake.patch
$popd$mkdir-p/opt/soci/socigit/build/head
$pushd/opt/soci/socigit/build/head
$cmake-DCMAKE_INSTALL_PREFIX=/usr-DCMAKE_BUILD_TYPE=Release-DSOCI_CXX11=ON-DSOCI_TESTS=OFF../..
$make
$sudomakeinstall
$popdMacOSConfigure SOCI:$mkdir-p/opt/soci/socigit/build/head&&cd/opt/soci/socigit/build/head
$cmake-DCMAKE_INSTALL_PREFIX=/usr/local\-DCMAKE_BUILD_TYPE=Debug-DSOCI_CXX11=ON\-DSOCI_ASAN=ON-DCMAKE_VERBOSE_MAKEFILE=OFF\-DSOCI_TESTS=OFF-DSOCI_STATIC=OFF-DSOCI_DB2=OFF-DSOCI_EMPTY=ON\-DSOCI_FIREBIRD=OFF-DSOCI_MYSQL=ON-DSOCI_ODBC=OFF-DSOCI_ORACLE=OFF\-DSOCI_POSTGRESQL=ON-DSOCI_SQLITE3=ON../..
$make
$sudomakeinstallBuilding the library and test binaryTo customize OpenTREP to your environment, you can alter
the installation directory:exportINSTALL_BASEDIR="${HOME}/dev/deliveries"exportTREP_VER="0.07.12"if[-d/usr/lib64];thenLIBSUFFIX="64";elseLIBSUFFIX="";fiexportLIBSUFFIX_4_CMAKE="-DLIB_SUFFIX=$LIBSUFFIX"Then, as usual:To configure the project, type something like:$rm-rfbuild&&mkdirbuild&&cdbuild
$cmake-DCMAKE_INSTALL_PREFIX=${INSTALL_BASEDIR}/opentrep-${TREP_VER}\-DCMAKE_BUILD_TYPE:STRING=Debug-DENABLE_TEST:BOOL=ON\-DINSTALL_DOC:BOOL=ON-DRUN_GCOV:BOOL=OFF${LIBSUFFIX_4_CMAKE}..With a specific version of Boost, sayBoost 1.69installed in a parallel way (for instance, as an optional module):$cmake-DCMAKE_INSTALL_PREFIX=${INSTALL_BASEDIR}/opentrep-${TREP_VER}\-DCMAKE_BUILD_TYPE:STRING=Debug-DENABLE_TEST:BOOL=ON\-DBOOST_LIBRARYDIR=/usr/lib64/boost169\-DBOOST_INCLUDEDIR=/usr/include/boost169\-DBoost_ADDITIONAL_VERSIONS="1.69 1.69.0"\-DINSTALL_DOC:BOOL=ON-DRUN_GCOV:BOOL=OFF${LIBSUFFIX_4_CMAKE}..For anad hocinstallation:INSTALL_DIR=/var/www/webapps/opentrep/trep

cmake-DCMAKE_INSTALL_PREFIX=${INSTALL_DIR}\-DCMAKE_BUILD_TYPE:STRING=Debug-DINSTALL_DOC:BOOL=OFF\-DRUN_GCOV:BOOL=OFF${LIBSUFFIX_4_CMAKE}..On MacOS, a few software (e.g., ICU and Readline) are not in
the standard place. So, thecmakecommand becomes:$exportCMAKE_CXX_FLAGS="-Wno-mismatched-new-delete";\cmake-DCMAKE_INSTALL_PREFIX=${INSTALL_BASEDIR}/opentrep-$TREP_VER\-DREADLINE_ROOT=/usr/local/opt/portable-readline\-DREADLINE_INCLUDE_DIR=/usr/local/opt/portable-readline/include\-DREADLINE_LIBRARY=/usr/local/opt/libedit/lib/libedit.dylib\-DICU_ROOT=/usr/local/opt/icu4c\-DCMAKE_BUILD_TYPE:STRING=Debug-DINSTALL_DOC:BOOL=ON\-DRUN_GCOV:BOOL=OFF${LIBSUFFIX_4_CMAKE}..To build the project, type:$makeTo test the project, type:$makecheckTo install the library (libopentrep*.so*) and the binary (opentrep),
just type:makeinstallcd${INSTALL_BASEDIR}rm-fopentrep-stable&&ln-sopentrep-${TREP_VER}opentrep-stablecd-To package the source files, type:$makedistTo package the binary and the documentation:$makepackageInstall the latestOpenTravelData (OPTD) POR data file.
Note that OpenTREP no longer ships with (full) OPTD data files;
onlytest filesare shipped.
The OPTD POR data file (optd_por_public_all.csv) has therefore
to be downloaded aside, usually renamed asoptd_por_public.csv,
and the OpenTREP binaries have then to be referred to the file-path
of that POR data file.$wget\https://github.com/opentraveldata/opentraveldata/raw/master/opentraveldata/optd_por_public_all.csv\-O${INSTALL_BASEDIR}/share/opentrep/data/por/optd_por_public.csvTo run the local binary version:$./opentrep/opentrep-{dbmgr,indexer,searcher}To run the installed version (the first following line must be done once
per session):$exportTREP_LIB="${INSTALL_BASEDIR}/opentrep-${TREP_VER}/lib${LIBSUFFIX}"$exportLD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${TREP_LIB}"$${INSTALL_BASEDIR}/opentrep-${TREP_VER}/bin/opentrep-{dbmgr,indexer,searcher}Underlying (relational) database, SQLite or MySQL/MariaDB, if anyOpenTREP may use, if so configured, a relational database. For now,
two database products are supported, SQLite3 and MySQL/MariaDB.
The database accelerates the look up of POR by (IATA, ICAO, FAA) codes
and of Geonames ID. When OpenTREP is configured to run without database,
those codes and Geonames ID are full-text searched directly with Xapian.
Note that the database can be managed directly,i.e., without the
OpenTREP search interface on top of it, thanks to theopentrep-dbmgrutility, which is detailed below.Use casesIndexing the POR dataFilling the (relational) database, SQLite or MySQL/MariaDB, if anyHere, for clarity reason, we use the local version. It is easy (see above)
to derive the same commands with the installed version.The following command prompts a Shell:$./opentrep/opentrep-dbmgr-tsqlite-p${INSTALL_BASEDIR}/share/opentrep/data/por/optd_por_public.csvThen, within theopentrep>shell, a typical sequence for SQLite would be:infocreate_userfill_from_por_filelist_nblist_by_iatancelist_by_icaolfmnlist_by_faaafmlist_by_geonameid6299418The following command prompts a shell:./opentrep/opentrep-dbmgr-tmysql-p${INSTALL_BASEDIR}/share/opentrep/data/por/optd_por_public.csvThen, within theopentrep>shell, a typical sequence for MySQL/MariaDB
would be:reset_connection_stringdb=mysqluser=rootpassword=<passwd>create_userreset_connection_stringdb=trep_trepuser=treppassword=trepfill_from_por_filelist_nblist_by_iatancelist_by_icaolfmnlist_by_faaafmlist_by_geonameid6299418Xapian indexing with standard installationBy default, the Xapian indexer runs without filling any relational database,
as that step can be performed independantly byopentrep-dbmgr,
as seen above.Xapian indexing without any relational database:$./opentrep/opentrep-indexer-p${INSTALL_BASEDIR}/share/opentrep/data/por/optd_por_public.csvXapian indexing and filling and indexing the SQLite database:$./opentrep/opentrep-indexer-tsqlite-p${INSTALL_BASEDIR}/share/opentrep/data/por/optd_por_public.csvXapian indexing and filling and indexing the MySQL/MariaDB database:$./opentrep/opentrep-indexer-tmysql-p${INSTALL_BASEDIR}/share/opentrep/data/por/optd_por_public.csvThere is an option to not even touch Xapian at all, for instance to check
that theOpenTravelData (OPTD) POR data fileis well-formed:$./opentrep/opentrep-indexer-x0-p${INSTALL_BASEDIR}/share/opentrep/data/por/optd_por_public.csvXapian indexing for an ad hoc deployed Web applicationXapian indexing without any relational database:$./opentrep/opentrep-indexer-d/var/www/webapps/opentrep/trep/traveldb-p${INSTALL_BASEDIR}/share/opentrep/data/por/optd_por_public.csv(Optional) Filling and indexing the SQLite database:$./opentrep/opentrep-dbmgr-tsqlite-s/var/www/webapps/opentrep/trep/sqlite_travel.db-p${INSTALL_BASEDIR}/share/opentrep/data/por/optd_por_public.csvcreate_userfill_from_por_filequitSearchingSearching without any relational database support:$./opentrep/opentrep-searcher-q"nce sfo"Searching with SQLite (note that it should be quicker than without a database):$./opentrep/opentrep-searcher-tsqlite-q"nce sfo"Searching with MySQL/MariaDB:$./opentrep/opentrep-searcher-tmysql-q"nce sfo"Searching with SQLite (with Xapian and SQLite DB in awebappsdirectory):$./opentrep/opentrep-searcher-d/var/www/webapps/opentrep/trep/traveldb-tsqlite-s/var/www/webapps/opentrep/trep/sqlite_travel.db-q"nce sfo"Deployment stagesThe idea is to have at least two pieces of infrastructure (SQL database,
Xapian index) in parallel:one is used by the production;the other one used as a staging platform in order to test
and validate a new version.Once the new version has been validated, the two pieces of infrastructure
can then be interverted, ie, the production becomes the new version,
and the older version ends up in staging.It means that all programs have to choose which version they want to work on.
That version may even be toggled in live.That method to deploy in production through a staging process
is even more needed by the fact that indexing a new POR data file
takes up to 30 minutes in the worst case. So, we cannot afford 30-45 minutes
of downtime everytime a new POR data file is released (potentially every day).With that staging process, it is even possible to fully automate
the re-indexing after a new POR data file release: once the new release
has been cleared byQA (Quality Assurance)on staging, it becomes production.The corresponding command-line option for the various programs
(opentrep-dbmgr,opentrep-indexer,opentrep-searcher) is-m.For instance, to build the Xapian index/database for deployment number 0:$./opentrep/opentrep-indexer-m0-d/var/www/webapps/opentrep/trep/traveldb
$chown-Rapache.apache/var/www/webapps/opentrep/trep
$ls-laFh/var/www/webapps/opentrep/trep/
-rw-r--r--1apacheapache16MOct142018sqlite_travel.db0
drwxr-xr-x2apacheapache4.0KOct142018traveldb0/And to build the Xapian index/database for deployment number 1:$./opentrep/opentrep-indexer-m1-d/var/www/webapps/opentrep/trep/traveldb$chown-Rapache.apache/var/www/webapps/opentrep/trep
$ls-laFh/var/www/webapps/opentrep/trep/
-rw-r--r--1apacheapache16MOct142018sqlite_travel.db1
drwxr-xr-x2apacheapache4.0KOct142018traveldb1/Index, or not, non-IATA PORThere is also a command-line option, namely-n, to state whether or not
the non-IATA-referenced POR should be included/parsed and indexed.By default, and historically, only the POR, which are referenced byIATA(ie, which have a specific IATA code), are indexed (and may be searched for)
in OpenTREP.POR are also referenced by other international organizations, such asICAOorUN/LOCODE,
and may not be referenced by IATA (in which case their IATA code
is left empty).As of October 2018, there are around 110,000 POR inOpenTravelData (OPTD),
the reference data source for OpenTREP:Around 20,000 POR are referenced by IATAAround 90,000 POR are not referenced by IATA, but referenced
by other international organizations (eg, ICAO, UN/LOCODE)Indexing 20,000 POR takes already a few minutes on standard hardware.Indexing 110,000 POR would take 15 to 20 minutes.Once indexed, all those POR become searchable. That flag is therefore
only used at indexing time (i.e., by theopentrep-dbmgrandopentrep-indexerprograms).Installing a Python virtual environmentAll the details are explained on adedicated procedure,
which works for the major Linux distributions and on MacOS.The procedure first installs a specific version of Python (as of January 2022,
3.9.9) thanks to PyEnv, then installpipenvthanks to thepiputility
provided with that specific Python version.Checking that the Python module worksWith the standard installation:PYTHON_VERSION=$(python--version2>&1|cut-d' '-f2,2|cut-d'.'-f1,2)PYTHONPATH=${INSTALL_BASEDIR}/opentrep-${TREP_VER}/lib${LIBSUFFIX}:${INSTALL_BASEDIR}/opentrep-${TREP_VER}/lib${LIBSUFFIX}/python${PYTHON_VERSION}/site-packages/pyopentrep\python3-c"import pyopentrep; \openTrepLibrary = pyopentrep.OpenTrepSearcher(); \initOK = openTrepLibrary.init (\"${INSTALL_BASEDIR}/opentrep-${TREP_VER}/share/opentrep/data/por/test_optd_por_public.csv\", '/tmp/opentrep/xapian_traveldb', 'sqlite', '/tmp/opentrep/sqlite_travel.db', 0, False, True, True, 'pyopentrep.log'); \print (openTrepLibrary.search ('S', 'los las'))"With anad hocinstallation:PYTHON_VERSION=$(python--version2>&1|cut-d' '-f2,2|cut-d'.'-f1,2)PYTHONPATH=${INSTALL_DIR}/lib${LIBSUFFIX}:${INSTALL_BASEDIR}/lib${LIBSUFFIX}/python${PYTHON_VERSION}/site-packages/pyopentrep\python3-c"import pyopentrep; \openTrepLibrary = pyopentrep.OpenTrepSearcher(); \initOK = openTrepLibrary.init (\"${INSTALL_BASEDIR}/opentrep-${TREP_VER}/share/opentrep/data/por/test_optd_por_public.csv\", '/var/www/webapps/opentrep/trep/traveldb', 'mysql', 'db=trep_trep user=trep password=trep', 0, False, True, True, 'pyopentrep.log'); \print (openTrepLibrary.search ('S', 'los las'))"Trouble-shooting Python issues on MacOSInterceptors not installed / lateOn some versions of MacOS (e.g., with 12.0 aka Monterey or 11.1 aka Big Sur),
there may be some strange issue related to interceptors:$./opentrep/python/pyopentrep-d/tmp/opentrep/xapian_traveldb"nce sfo"==217==ERROR:Interceptorsarenotworking.ThismaybebecauseAddressSanitizerisloadedtoolate(e.g.viadlopen).Pleaselaunchtheexecutablewith:DYLD_INSERT_LIBRARIES=/Library/Developer/CommandLineTools/usr/lib/clang/12.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib"interceptors not installed"&&0Aborttrap:6A work around is to explicitly use the MacOS native Python interpreter.
The whole command-line then becomes:$PYTHON_VERSION=$(python3--version2>&1|cut-d' '-f2,2|cut-d'.'-f1,2)$PYTHONPATH=${INSTALL_BASEDIR}/opentrep-${TREP_VER}/lib${LIBSUFFIX}:${INSTALL_BASEDIR}/opentrep-${TREP_VER}/lib${LIBSUFFIX}/python${PYTHON_VERSION}/site-packages/pyopentrep\DYLD_INSERT_LIBRARIES=/Library/Developer/CommandLineTools/usr/lib/clang/12.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib\ASAN_OPTIONS=detect_container_overflow=0\/usr/local/Cellar/python/3.9.1/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python\./opentrep/python/pyopentrep.py-d/tmp/opentrep/xapian_traveldb"nce sfo"OPTD-maintainedlistofPOR(pointsofreference):'~/dev/deliveries/opentrep-${TREP_VER}/share/opentrep/data/por/test_optd_por_public.csv'Xapian-basedtraveldatabase/index:'/tmp/opentrep/xapian_traveldb0'SQLitedatabase:'/tmp/opentrep/sqlite_travel.db'searchString:ncesfo
Compactformat=>recognisedplace(city/airport)codes:
NCESFO
------------------OpenTREP as a Python extensionReferencesScikit-build (packaging C/C++ Python extensions):https://scikit-build.readthedocs.io/en/latest/https://scikit-build.readthedocs.io/en/latest/usage.html#basic-usage-examplePyJQ (not using Scikit-build):https://github.com/doloopwhile/pyjqBuild and package OpenTREP as a Python extensionOpenTREP depends on Boost libraries, including Boost Python C extensions,
and on Protobuf Python extensions too. Those Boost and Protobuf C extensions
generally come with the system (e.g., installed by Homebrew on MacOS,
DNF on Fedora/CentOS/RedHat, APT on Debian/Ubuntu).Hence, installing a local virtual environment will not work,
especially when the Python version of the virtual environment does not
match exactly the Python version of the system-installed libraries.The simplest approach so far is to use the Python installed by the system.
Still, this can be done withpip, with that latter potentially installed
on the user account only with the--useroption when upgradingpipitself (withpython -m pip install --user -U pip).Note that Linux binary wheels cannot be pushed as is onto PyPi.Manylinuxshould be used for that.
Scikit-build maintainssome additions on top of Manylinuxand thecorresponding Manylinux Docker images.
The way to run those have still to be documented below.
In the meantime, the OpenTREP wheel has to be built from the sources, either
(see below for the details):By using thepipmodule of the system-based Python:By cloningthis OpenTREP Git repositoryInstall Python modules/dependenciesInstall Python 3 modules if not already done so:$pyenvglobalsystem
$python-mpipinstall--user-Upip
$python-mpipinstall-Usetuptoolscmakebuildwheelninjascikit-build
$python-mpipinstall-Upytesttoxtwinesphinxkeyrings.altflake8black
$python-mpipinstall-Usimplejsonprotobuf
$python-mpipinstall-UpyjqpyyamlInstall OpenTrep Python extension with system-basedpipOpenTREP extension/module is installed here as a system-based module
with system-based Python andpipBuild from source and install withpip:$python-mpipinstall-Uopentrep
Defaultingtouserinstallationbecausenormalsite-packagesisnotwriteable
CollectingopentrepUsingcachedopentrep-0.7.10.tar.gz(1.7MB)Installingbuilddependencies...doneGettingrequirementstobuildwheel...donePreparingwheelmetadata...doneBuildingwheelsforcollectedpackages:opentrepBuildingwheelforopentrep(PEP517)...doneCreatedwheelforopentrep:filename=opentrep-0.7.10-cp39-cp39-linux_x86_64.whlsize=3060796sha256=6362e3a86af016b251fe33b9f76db17322ec15a60575082f31f6b719ba2cf97fStoredindirectory:~/.cache/pip/wheels/82/b3/7c/f026b883cc204eefab1588f5e68661f78fec25395277bd221d
Successfullybuiltopentrep
Installingcollectedpackages:opentrep
Successfullyinstalledopentrep-0.7.10Set theLD_LIBRARY_PATH/DYLD_LIBRARY_PATHandPYTHONPATHenvironment variablesOn Linux:$INST_DIR=${HOME}/.localTREPBINDIR=${INST_DIR}/binOPTDPOR=${INST_DIR}/share/opentrep/data/por/test_optd_por_public.csvexportLD_LIBRARY_PATH=${INST_DIR}/libexportPYTHONPATH=${INST_DIR}/lib:${INST_DIR}/lib/python3.9/site-packages/pyopentrepOn MacOS:$exportINST_DIR=/usr/localexportPYTHONPATH=${INST_DIR}/lib:${INST_DIR}/lib/python3.9/site-packages/pyopentrepexportDYLD_LIBRARY_PATH=${INST_DIR}/libSee how to use the newly installed extension
in the dedicated sub-section belowBuild the OpenTrep Python extension locally with system-based Scikit-buildSetPyenvon the system-based Python, which should be Python 3:$pyenvglobalsystem
$python--version
Python3.9.9Clean potential former builds and launch a new build with Scikit-build:$rm-rf_skbuild/dist/.tox/__pycache__/.pytest_cache/MANIFEST*.egg-info/
$pythonsetup.py--build-type=Debugbuildsdistbdist_wheel# the build takes a few minutes$ls-lFh_skbuild/*-x86_64-3.9/dist/
dist/:
total24408-rw-r--r--1userstaff9.6MJan1619:10opentrep-0.7.10.post2-cp39-cp39-macosx_12_0_x86_64.whl
-rw-r--r--1userstaff1.6MJan1619:10opentrep-0.7.10.post2.tar.gz

_skbuild/*-x86_64-3.9/:
-rw-r--r--1userstaff0BJan1019:10_skbuild_MANIFEST
drwxr-xr-x24userstaff768BJan1019:10cmake-build/
drwxr-xr-x6userstaff192BJan1019:10cmake-install/
drwxr-xr-x3userstaff96BJan1019:10setuptools/Set theLD_LIBRARY_PATHandPYTHONPATHenvironment variables:$INST_DIR=${PWD}/_skbuild/macosx-12.0-x86_64-3.9/cmake-installTREPBINDIR=${INST_DIR}/binOPTDPOR=${INST_DIR}/share/opentrep/data/por/test_optd_por_public.csvexportLD_LIBRARY_PATH=${INST_DIR}/libexportPYTHONPATH=${INST_DIR}/lib:${INST_DIR}/lib/python3.9/site-packages/pyopentrepManylinux (as of mid-2020, limited to Python2, so, not working):$dockerpullscikitbuild/manylinux2010_x86_64:09d11d5f8
$dockerrun--rm-ePLAT=manylinux2010_x86_64-v`pwd`:/ioscikitbuild/manylinux2010_x86_64:09d11d5f8linux64/io/travis/build-wheels.shUpload to test PyPi (remember, no Linux binary wheel):user@laptop$PYPIURL="https://test.pypi.org"user@laptop$twineupload-u__token__--repository-url${PYPIURL}/legacy/dist/*
Uploadingdistributionstohttps://test.pypi.org/legacy/
Uploadingopentrep-0.7.10-cp39-cp39-macosx_12_0_x86_64.whl100%|█████████████████████████████████████████████████████████████████████|9.86M/9.86M[01:00<00:00,172kB/s]Uploadingopentrep-0.7.10.tar.gz100%|█████████████████████████████████████████████████████████████████████|1.65M/1.65M[00:12<00:00,139kB/s]Viewat:
https://test.pypi.org/project/opentrep/0.7.10/Upload/release the Python packages onto thePyPi repository:user@laptop$PYPIURL="https://upload.pypi.org"user@laptop$keyringset${PYPIURL}/__token__
Passwordfor'__token__'in'${PYPIURL}/':
user@laptop$twineupload-u__token__--repository-url${PYPIURL}/legacy/dist/*
Uploadingdistributionstohttps://upload.pypi.org/legacy/
Uploadingopentrep-0.7.10.post2-cp39-cp39-macosx_12_0_x86_64.whl100%|█████████████████████████████████████████████████████████████████████|9.86M/9.86M[01:00<00:00,172kB/s]Uploadingopentrep-0.7.10.post2.tar.gz100%|█████████████████████████████████████████████████████████████████████|1.65M/1.65M[00:12<00:00,139kB/s]Viewat:
https://pypi.org/project/opentrep/0.7.10.post2/Test the OpenTREP Python extensionJust to be sure the local configuration does not interfere,
set the version of Python to be the system oneOn most recent distributions, it would be 3.10.x:$pyenvlocalsystem
$python-V
Python3.10.1
$typepython3
python3is~/.pyenv/shims/python3+ On MacOS:$pyenvlocalsystem
$python3-V
Python3.9.9
$typepython3
python3ishashed(/usr/local/opt/python@3.9/bin/python3)Remove older version of and install theopentrepPython extensionOn Linux:$python-mpipuninstallopentrep
$python-mpipinstall-Uopentrep# will take a few minutes, as OpenTREP is built againOn MacOS
(noramlly,python3should point to/usr/local/Cellar/python\@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python):$python3-mpipuninstallopentrep
$python3-mpipinstall-UopentrepCheck that the OpenTREP Python extension has been installedOn Linux:$python-mpipshowopentrep
Name:opentrep
Version:0.7.10.post2
...
Location:~/.pyenv/versions/3.10.1/lib/python3.9/site-packages
Requires:protobufOn MacOS:$python3-mpipshowopentrep
Name:opentrep
Version:0.7.10.post2
...
Location:/usr/local/lib/python3.9/site-packages
Requires:protobufSet thePYTHONPATHenvironment variable to where the system Python
is installedOn Linux:$exportPYTHONPATH="${HOME}/.local/lib:${HOME}/.local/lib/python3.10/site-packages/pyopentrep"exportLD_LIBRARY_PATH="${HOME}/.local/lib:${HOME}/.local/lib/python3.10/site-packages/pyopentrep:${LD_LIBRARY_PATH}"On MacOS (Python binary is/usr/local/Cellar/python\@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Pythonand Python libraries are installed simply in/usr/local/lib):$exportPYTHONPATH="/usr/local/lib:/usr/local/lib/python3.9/site-packages/pyopentrep"Launch a simple end-to-end test withpytestOn Linux:$python-mpytesttest_trep_e2e_simple.pyOn MacOS:$DYLD_INSERT_LIBRARIES=/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylibASAN_OPTIONS=detect_container_overflow=0/usr/local/Cellar/python\@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python-mpytesttest_trep_e2e_simple.pyIt should give something like:===============testsessionstarts============platformdarwin--Python3.9.9,pytest-6.2.5,py-1.11.0,pluggy-1.0.0
rootdir:$HOME/dev/geo/opentrep
collected1item

test_trep_e2e_simple.py[100%]==================1passedin1.38s==============Use the OpenTREP Python extensionDownload the latest OpenTravelData (OPTD) POR data fileIf not already done, install a few more Python modulesOn Linux:$python-mpipinstall-UopentrepwrapperopentraveldataOn MacOS:$python3-mpipinstall-UopentrepwrapperopentraveldataDownload and use the latest POR data fileOn Linux:$python
Python3.9.9(default,November2021)On MacOS:$DYLD_INSERT_LIBRARIES=/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylibASAN_OPTIONS=detect_container_overflow=0/usr/local/Cellar/python\@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/PythonPython interactive shell:>>>importopentraveldata>>>myOPTD=opentraveldata.OpenTravelData()>>>myOPTD.downloadFilesIfNeeded()>>>myOPTDOpenTravelData:LocalIATA/ICAOPORfile:/tmp/opentraveldata/optd_por_public_all.csvLocalUN/LOCODEPORfile:/tmp/opentraveldata/optd_por_unlc.csv>>>myOPTD.fileSizes()(44588871,4967092)Xapian index initializationInitialize the Xapian index with the-ioption ofpyopentrep.py,
so as to index the full OpenTravelData (OPTD) POR (points of reference)
data fileOn Linux:$python~/.local/lib/python3.10/site-packages/pyopentrep/pyopentrep.py-p/tmp/opentraveldata/optd_por_public_all.csv-iOn MacOS:$DYLD_INSERT_LIBRARIES=/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylibASAN_OPTIONS=detect_container_overflow=0/usr/local/Cellar/python\@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python/usr/local/lib/python3.9/site-packages/pyopentrep/pyopentrep.py-p/tmp/opentraveldata/optd_por_public_all.csv-iOPTD-maintainedlistofPOR(pointsofreference):'/tmp/opentraveldata/optd_por_public_all.csv'Xapian-basedtraveldatabase/index:'/tmp/opentrep/xapian_traveldb0'SQLitedatabase:'/tmp/opentrep/sqlite_travel.db'Performtheindexationofthe(Xapian-based)traveldatabase.
Thatoperationmaytakeseveralminutesonsomeslowmachines.
Ittakeslessthan20secondsonfastones...
Numberofactuallyparsedrecords:1,000,outof103,394recordsinthePORdatafilesofar
...
Numberofactuallyparsedrecords:20,000,outof122,394recordsinthePORdatafilesofar
Done.Indexed20372POR(pointsofreference)The Xapian index may also be initialized with the C++ (non-Python)opentrep-indexerutility (that is the former way of initializing
the Xapian index, when it was not available from the Python utility):$${TREPBINDIR}/opentrep-indexer-tsqlite-a1-p${OPTDPOR}PORfile-pathis:~/.pyenv/versions/3.9.9/share/opentrep/data/por/test_optd_por_public.csv
Deploymentnumber:0Xapianindex/databasefilepathis:/tmp/opentrep/xapian_traveldb0
SQLdatabasetypeis:sqlite
SQLdatabaseconnectionstringis:/tmp/opentrep/sqlite_travel.db0
Arenon-IATA-referencedPORincluded?0IndexthePORinXapian?1Addandre-indexthePORintheSQL-baseddatabase?1Logfilenameis:opentrep-indexer.log
ParsingandindexingtheOpenTravelDataPORdatafile(intoXapianand/orSQLdatabases)maytakeafewtensofminutesonsomearchitectures(andafewminutesonfastestones)...9entrieshavebeenprocessedSearch with the OpenTrep Python extensionUse a Python wrapper script around the OpenTrep Python extension to search
for termsOn Linux:$python~/.local/lib/python3.10/site-packages/pyopentrep/pyopentrep.pyOn MacOS:$DYLD_INSERT_LIBRARIES=/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylibASAN_OPTIONS=detect_container_overflow=0/usr/local/Cellar/python\@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python/usr/local/lib/python3.9/site-packages/pyopentrep/pyopentrep.pyOPTD-maintainedlistofPOR(pointsofreference):'/tmp/opentrep/test_optd_por_public.csv'Xapian-basedtraveldatabase/index:'/tmp/opentrep/xapian_traveldb0'SQLitedatabase:'/tmp/opentrep/sqlite_travel.db'searchString:snafranciscoriodejanerolosangelesreykyavki
Compactformat=>recognisedplace(city/airport)codes:
SFORIOLAXREK
------------------When the full POR data file has been indexedOn Linux:$python~/.local/lib/python3.10/site-packages/pyopentrep/pyopentrep.py-fF"cnsha deham deess"On MacOS:$DYLD_INSERT_LIBRARIES=/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylibASAN_OPTIONS=detect_container_overflow=0/usr/local/Cellar/python\@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python/usr/local/lib/python3.9/site-packages/pyopentrep/pyopentrep.py-fF"cnsha deham deess"OPTD-maintainedlistofPOR(pointsofreference):'/tmp/opentrep/test_optd_por_public.csv'Xapian-basedtraveldatabase/index:'/tmp/opentrep/xapian_traveldb0'SQLitedatabase:'/tmp/opentrep/sqlite_travel.db'searchString:cnshadehamdeess
RawresultfromtheOpenTreplibrary:1.SHA-A-6301388,34.7007%,ShanghaiHongqiaoInternationalAirport,ShanghaiHongqiaoInternationalAirport,ZSSS,,CNSHA,,0,1970-Jan-01,2999-Dec-31,,SHA|1796236|Shanghai|Shanghai|CN|SH,SH,CN,,China,713,China,CNY,NA,Asia,31.1979,121.336,S,AIRP,23,Shanghai,Shanghai,,,,Z,,0,3,3,Asia/Shanghai,8,8,8,2014-Aug-01,,https://en.wikipedia.org/wiki/Shanghai_Hongqiao_International_Airport,31.1979,121.336,cnsha,cnsha,34.7007%,0,02.HAM-C-2911298,12.8103%,Hamburg,Hamburg,,,DEHAM,,0,1970-Jan-01,2999-Dec-31,,HAM|2911298|Hamburg|Hamburg|DE|HH,HH,DE,,Germany,429,Germany,EUR,NA,Europe,53.5753,10.0153,P,PPLA,04,Hamburg,Hamburg,00,,,02000,02000000,1739117,0,2,Europe/Berlin,1,2,1,2019-Nov-28,HAM,LBC,OBZ,XFW,ZMB,https://en.wikipedia.org/wiki/Hamburg,53.5507,9.99302,deham,deham,12.8103%,0,03.ESS-C-2928810,1.34094%,Essen,Essen,,,DEESS,,0,1970-Jan-01,2999-Dec-31,,ESS|2928810|Essen|Essen|DE|NW,NW,DE,,Germany,429,Germany,EUR,NA,Europe,51.4566,7.01228,P,PPLA3,07,NorthRhine-Westphalia,NorthRhine-Westphalia,051,DüsseldorfDistrict,DuesseldorfDistrict,05113,05113000,593085,0,83,Europe/Berlin,1,2,1,2019-Oct-22,ESS,ESZ,https://en.wikipedia.org/wiki/Essen,0,0,deess,deess,1.34094%,0,0------------------Use the OpenTrep Python extension from a Python interactive shellOn Linux:$python
Python3.10.1(main,Jan102022,...)onlinuxOn MacOS:$DYLD_INSERT_LIBRARIES=/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylibASAN_OPTIONS=detect_container_overflow=0/usr/local/Cellar/python\@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python>>>importpyopentrep>>>openTrepLibrary=pyopentrep.OpenTrepSearcher()>>>initOK=openTrepLibrary.init('/tmp/opentraveldata/optd_por_public.csv','/tmp/opentrep/xapian_traveldb','sqlite','/tmp/opentrep/sqlite_travel.db',0,False,True,True,'pyopentrep.log')>>>openTrepLibrary.search('S','nce sfo')'NCE/0,SFO/0'>>>openTrepLibrary.search('F','nce sfo')"1. NCE-C-2990440, 8.16788%, Nice, Nice, , , FRNCE, , 0, 1970-Jan-01, 2999-Dec-31, , NCE|2990440|Nice|Nice|FR|PAC, PAC, FR, , France, 427, France, EUR, NA, Europe, 43.7031, 7.26608, P, PPLA2, 93, Provence-Alpes-Côte d'Azur, Provence-Alpes-Cote d'Azur, 06, Alpes-Maritimes, Alpes-Maritimes, 062, 06088, 338620, 25, 18, Europe/Paris, 1, 2, 1, 2019-Sep-05, NCE, https://en.wikipedia.org/wiki/Nice, 0, 0, NA, nce, 0%, 0, 0\n2. SFO-C-5391959, 32.496%, San Francisco, San Francisco, , , USSFO, , 0, 1970-Jan-01, 2999-Dec-31, , SFO|5391959|San Francisco|San Francisco|US|CA, CA, US, , United States, 91, California, USD, NA, North America, 37.7749, -122.419, P, PPLA2, CA, California, California, 075, City and County of San Francisco, City and County of San Francisco, Z, , 864816, 16, 28, America/Los_Angeles, -8, -7, -8, 2019-Sep-05, SFO, https://en.wikipedia.org/wiki/San_Francisco, 0, 0, NA, sfo, 0%, 0, 0\n">>>quit()Search with the OpenTrepWrapper packageUse the OpenTREP wrapper on the full indexOn Linux:$python
Python3.10.1(main,Jan102022,...)onlinuxOn MacOS:$DYLD_INSERT_LIBRARIES=/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylibASAN_OPTIONS=detect_container_overflow=0/usr/local/Cellar/python\@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/PythonPython interactive shell:>>>fromOpenTrepWrapperimportmain_trep,index_trep>>>main_trep(searchString='nce sfo',outputFormat='S',xapianDBPath='/tmp/opentrep/xapian_traveldb',logFilePath='/tmp/opentrep/opeentrep-searcher.log',verbose=False)([(173.579,'NCE'),(390.644,'SFO')],'')>>>main_trep(searchString='cnsha deham',outputFormat='S',xapianDBPath='/tmp/opentrep/xapian_traveldb',logFilePath='/tmp/opentrep/opeentrep-searcher.log',verbose=False)([(0.34700699999999995,'SHA'),(0.128103,'HAM')],'')>>>quit()(Optional) Running the Django-based application server (needs update)Install Django as a Python moduleOn Linux:$python-mpipinstall-UdjangoOn MacOS:$python3-mpipinstall-UdjangoGo in the Django directory:$exportTREP_TRAVELDB=/tmp/opentrep/traveldb
$exportTREP_LOG=django_trep.log
$pushdgui/django/webapps/opentrepStart the Django Web application serverOn Linux:$pythonmanage.pyrunserverlocalhost:8000
Python3.10.1(default,November2021)On MacOS (as of end 2021, does not work):$DYLD_INSERT_LIBRARIES=/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylibASAN_OPTIONS=detect_container_overflow=0/usr/local/Cellar/python\@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Pythonmanage.pyrunserverlocalhost:8000Query OpenTREP with a web browser:$open"http://localhost:8000/search?q=rio de janero reykyavik sna francicso"Go back to the top directory:$popd