Package: fastfilepackage
Fast File PackageAn module written with pure Python C Extensions to open a file and
cache the more recent accessed lines.
See these files for an usage example:tests/fastfileperformance.pytests/fastfiletest.pytests/getline_c_performance.cpptests/getline_cpp_performance.cppInstallationRequiresPython 3,pip3,distutils,setuptools,wheeland
aC++ 11compiler installed:sudo apt-get install build-essential g++ python3-dev python3 python3-pip
pip3 install setuptools wheelThen, you can clone this repository with:git clone https://github.com/evandrocoan/fastfilepackage
cd fastfilepackage
pip3 install .Or install it with:pip3 install fastfilepackageAlternive file readingThere are available 3 alternative implementations for file reading.
Both of them,
should have the same performance.You can defineFASTFILE_GETLINE=0to use the Python builtins.open() implementation (default)You can defineFASTFILE_GETLINE=1to use the C++ std::getline() implementationYou can defineFASTFILE_GETLINE=2to use the POSIX C getline() implementationUsage examples:FASTFILE_GETLINE=1 pip3 install . -vFASTFILE_GETLINE=1 FASTFILE_DEBUG=1 pip3 install . -vFile reading optimizationsYou can enable a file reading optimization with the environment variableFASTFILE_REGEX=1.
Do not define this variable or define it asFASTFILE_REGEX=0to disable the optimization.FASTFILE_REGEX=1will use the C language builtin regex libraryregex.h:https://linux.die.net/man/3/regexecFASTFILE_REGEX=2will use PCRE2 regex librarypcre2.h:http://pcre.org/current/doc/html/pcre2_match.html,
it requires the installation of thelibpcre2-devpackage withsudo apt-get install libpcre2-devFASTFILE_REGEX=3will use RE2 regex libraryre2/re2.h:https://github.com/google/re2/wiki/CplusplusAPI,
it requires the installation of there2library with:gitclonehttps://github.com/google/re2re2&&cdre2&&make&&sudomakeinstall&&maketestinstallFASTFILE_REGEX=4will use Hyperscan regex libraryhs.h:https://github.com/intel/hyperscan,
it requires the installation of thelibhyperscan-devpackage withsudo apt-get install libhyperscan-devNotes:Defining the variableFASTFILE_REGEXonly has effect whenFASTFILE_GETLINE=2is set (as/with value 2).
If the variableFASTFILE_GETLINE=2is not defined (as/with value 2),
any definition ofFASTFILE_REGEXis ignored.DebuggingYou you use theFASTFILE_DEBUG=1variable specified on theEnable debug modesection,
you do not need to build the program withCFLAGS="-O0..." CXXFLAGS="-O0 ..." /usr/bin/pip3 install .because these flags are already set byFASTFILE_DEBUG=1.If Python got segmentation fault,
you need to install the debug symbol packages,
and compile the program into mode debug.
These instructions works for both Linux and
Cygwin.sudo apt-get install python3-dbg
apt-cyg install libcrypt-devel python36-devel python36-debuginfo python3-debuginfo python3-cython-debuginfo
CFLAGS="-O0 -g -ggdb -fstack-protector-all" CXXFLAGS="-O0 -g -ggdb -fstack-protector-all" /usr/bin/pip3 install .

cd modulename/source
gdb --args /usr/bin/python3 -m modulename.__init__ -p ../tests/light/
python3 -m pdb -m modulename.__init__ -p ../tests/light/

cd /usr/bin
/usr/bin/python3 -u -m trace -t modulename-script.py -p ../tests/light/https://github.com/spiside/pdb-tutorialhttps://docs.python.org/3.7/library/pdb.htmlhttps://docs.python.org/3.7/library/trace.htmlhttps://stackoverflow.com/questions/1629685/when-and-how-to-use-gccs-stack-protection-featurehttps://stackoverflow.com/questions/25678978/how-to-debug-python-script-that-is-crashing-pythonhttps://stackoverflow.com/questions/46265835/how-to-debug-a-python-module-run-with-python-m-from-the-command-lineIn case you program enter on a deadlock (or livelock),
you can usegdbto discover or what is happening.
For this,
first you need get acore dump fileof the system somehow.
Once you get thecore dumpfile,
you can callgdb program_name coredump.
Now,
you can use thegdbcommands to navigate between the existing threads and
to discover which of them are waiting for the other,
i.e.,e it is causing the deadlock or livelock.p / s 0x6ffffdf8ed0print or content on the address with the format "%s"x / 100w 0x6ffffdf8ed0"examines" the specified memory addressframe 0shows the corresponding stack frame0asbt f(backtrace full)bt f 5shows the last 5 stack frames with all debugging symbols dataf 0andp varnameprint thevarnameonframe 0contexthttps://stackoverflow.com/questions/14659147/how-to-print-pointer-content-in-gdbhttps://stackoverflow.com/questions/14502236/how-to-view-a-pointer-like-an-array-in-gdbNote:
Instead of creating acore dump file,
you can run your program directly withgdb --args program_nameand
once you are on thegdbcommand line,
you can use theruncommand and
once your program enters/starts a dead or live lock,
you can pressCtrl+Cto stop the program execution.
Then,gdbwill already had "captured" thecore dum file.Enable debug modeThe default debug level is0where no debugging message code is generated into the final binary.
Guaranteeing maximum performance.
If you would like to compile a binary with debug messages,
define the environment variableFASTFILE_DEBUGbefore running the installer.
You can see the debugging level available on the filesource/debugger.h:FASTFILE_DEBUG=1 pip install .
FASTFILE_DEBUG=1 pip install -e .

FASTFILE_DEBUG=1 python setup.py install
FASTFILE_DEBUG=1 python setup.py developIf you are on Windows,
run it like this:set "FASTFILE_DEBUG=1" && pip install .
set "FASTFILE_DEBUG=1" && pip install -e .
set "FASTFILE_DEBUG=1" && python setup.py install
set "FASTFILE_DEBUG=1" && python setup.py developTo debugrefcountsleaks:CPython compiled with./configure --with-pydebugRuntime compiled with C assertion:
crash (kill itself with SIGABRT signal) if a C assertion fails (assert(...);).Use the debug hooks on memory allocators by default,
asPYTHONDEBUG=debugenvironment variable:
detect memory under and overflow and
misuse of memory allocators.Compiled without compiler optimizations (-Ogor even-O0) to be usable with a debugger likegdb:python-gdb.pyshould work perfectly.
However,
the regular runtime is unusable withgdbsince most variables and
function arguments are stored in registers,
and sogdbfails with the<optimized out>message.https://pythoncapi.readthedocs.io/runtimes.html#debug-buildhttps://stackoverflow.com/questions/40121749/why-gcc-og-doesnt-generate-source-code-line-mappinghttps://stackoverflow.com/questions/8650972/how-do-i-debug-refcounts-in-a-python-c-extension-the-easiest-wayTo generate core dumps instead of stack traces onCygwinexport CYGWIN="$CYGWIN error_start=dumper -d %1 %2"https://stackoverflow.com/questions/320001/using-a-stackdump-from-cygwin-executableIf you see this when runninggdb:[New Thread 8980.0x626c]
[New Thread 8980.0x1ba0]
[New Thread 8980.0x2454]
[New Thread 8980.0x3fbc]
warning: the debug information found in "/usr/lib/debug//usr/bin/cygwin1.dbg" does not match "/usr/bin/cygwin1.dll" (CRC mismatch).

[New Thread 8980.0x21c4]Run the commandrm /usr/lib/debug//usr/bin/cygwin1.dbghttp://cygwin.1069669.n5.nabble.com/Debugging-with-GDB-td94421.htmlLicenseSee the fileLICENSE.txt