Package: calmjs.rjs
calmjs.rjsPackage for the integration ofRequireJSinto a Python environment
via theCalmjs framework, to provide a reproducible workflow for the
generation of deployable artifacts from JavaScript source code provided
by Python packages in conjunction with standard JavaScript orNode.jspackages sourced fromnpmor other similar package repositories.IntroductionWeb applications can be created using any language, however the
interactive front-end user interfaces they provide ultimately rely on
some form of JavaScript.  Python web application frameworks or systems
that provide frontend functionalities that bridge the backend have
adopted the usage ofNode.jsfor testing the JavaScript code that
they may provide, withnpm(orbower) being the package manager
for the acquisition of JavaScript packages required for the associated
functionality.  This often resulted in the separation of what would have
been a single set of package dependency configuration into multiple
different sets; often this also resulted in the package being fractured
into two parts to fit in with the distribution channels being used (PyPI
vs npm and others).This outcome ends up being problematic for Python package management due
to the increase in difficulty in the propagation of the package’s
version and dependency information across all relevant package
management channels in a consistent, portable and reproducible manner
for downstream packages and their users.  The other issue is that the
configuration files used for asset management or artifact generation is
often coupled tightly to the system at hand, making it rather difficult
for their downstream package to reuse these configurations to generate
a combined artifacts that work also with their other upstream packages
in a consistent manner.Some other package managers attempt to solve this by being utterly
generic, however they lack the awareness of locally available Python
packages (such as Python wheels already installed in the local
environment not being understood by Bower), thus build processes that
involve Bower often end up relying on public infrastructure, and options
to move it to a private infrastructure or even reuse locally available
artifacts/packages require extra configurations which negate the
benefits offered by these systems.  Also, these build scripts are
tightly coupled to a specific project which are not portable.The goal of the Calmjs framework is to bring this separation back
together by providing the method to expose JavaScript sources included
with Python packages, with this package,calmjs.rjs, to provide the
facilities to produce deployable artifacts from those exported source
files, plus the other declared external bundles to be sourced fromnpmor other related Node.js package management systems.FeaturesHowcalmjs.rjsworksThe Calmjs framework provides the framework to allow Python packages to
declare the dependencies they need againstnpmbased packages for the
JavaScript code they provide, and also the system that allow Python
packages to declare which of their modules export JavaScript sources
that can be reused.The utility included withcalmjs.rjsprovide the means to consume
those declarations, treating the JavaScript files as both source and
compilation target, with the final deployable artifact(s) being produced
throughr.jsfrom therequirejspackage.Currently, the source files could be written in both AMD and CommonJS
module formats, although the CommonJS format is recommended due to their
wide support under most systems, and thatcalmjs.rjsprovides
transpilation and configuration generation utilities that processes the
JavaScript source code into a form that is compatible with ther.jsoptimizer.  However, theexportsstatement in the source file should
be not part ofmodule.exportsfor the mean time.  The AMD headers
and footers can be absent too, as thecalmjs rjstranspiler will add
the appropriate headers and footers needed (for example, haverequirebe imported from the correct source, or for mappingexportstomodule.exports) so that the final script will be
usable for the target platform or format.The resulting sources will be placed in a build directory, along with
all the declared bundled sources acquired from the Node.js package
managers or repositories.  A build file will then be generated that will
include all the relevant sources as selected to enable the generation of
the final artifact file throughr.js.  These can then be deployed to
the appropriate environment, or the whole above process can be included
as part of the functionality of the Python backend at hand.Ultimately, the goal ofcalmjs.rjsis to ease the integration and
interactions between of client-side JavaScript with server-side Python,
by simplifying the task of building, shipping and deployment of the two
set of sources in one shared package and environment.  The Calmjs
framework provides the linkage between these two environment and the
tools provided by there will assist with the setup of a common,
reproducible local Node.js environments.Finally, for quality control, this package has integration withcalmjs.dev, which provides the tools needed to set up the test
environment and harnesses for running of JavaScript tests that are part
of the Python packages for the associated JavaScript code.  However,
that package is not declared as a direct dependency, as not all use
cases will require the availability of that package.  Please refer to
installation section for details.Do note, in the initial implementation, the JavaScript source file
supported by this framework loosely follows certain definitions that
only mimic what ES6 intends to provide (as outlined earlier).  Even with
this, as a consequence of treating JavaScript within the Python package
as a source file for the compilation target which is the deployable
artifact file, the input source files and exported paths generated bycalmjs.rjsare NOT meant for direct consumption of web clients such as
web browsers.  The produced artifact from this framework will be usable
through the AMD API.InstallationIt is recommended that the local environment already have Node.js andnpminstalled at the very minimum to enable the installation ofrequirejs, if it hasn’t already been installed and available.  Also,
the version of Python must be either 2.7 or 3.3+; PyPy is supported,
with PyPy3 version 5.2.0-alpha1 must be used due to a upstream package
failing to function in the currently stable PyPy3 version 2.4.To installcalmjs.rjsinto a given Python environment, it may be
installed directly from PyPI with the following command:$pipinstallcalmjs.rjsInstalling/using RequireJS with calmjsTo establish a development/build environment for a Python package with
the support forr.jsthroughcalmjs.rjsin the current working
directory (e.g. for a project), the following command may be executed:$calmjsnpm--installcalmjs.rjsWhile runningnpm install requirejs(along with other related
packages declared bycalmjs.rjsthat it needs fromnpm) will achieve
the same effect, do note the Calmjs framework makes it possible fornpmdependencies to be propagated down to dependent packages; such
that if a Python package that have declaredcalmjs.rjsas a dependency
(either throughinstall_requiresor anextras_requirein itssetup.py) may have its complete set of dependencies onnpmbe
installed using the following command (assuming the package is namedexample.package:$calmjsnpm--installexample.packageAll standard JavaScript and Node.js dependencies forexample.packagewill now be installed into the current directory through the relevant
tools.  This process will also install all the other dependencies
throughnpmorrequirejsthat other Python packages depended on byexample.packagehave declared.For further details about how this all works can be found in the
documentation forcalmjs.  Otherwise, please continue on to theusagesection.Alternative installation methods (advanced users)Development is still ongoing withcalmjs.rjs, for the latest features
and bug fixes, the development version can be installed through git like
so:$pipinstallcalmjs$pipinstallgit+https://github.com/calmjs/calmjs.rjs.git#egg=calmjs.rjsAlternatively, the git repository can be cloned directly and executepython setup.py developwhile inside the root of the source
directory.Keep in mind thatcalmjsMUST be available before thesetup.pywithin thecalmjs.rjssource tree is executed, for it needs thepackage_jsonwriting capabilities incalmjs.  Alternatively,
please executepython setup.py egg_infoif any message aboutUnknown distribution option:is noted during the invocation ofsetup.py.Ascalmjsis declared as both namespace and package, there are certain
low-level setup that is required on the working Python environment to
ensure that all modules within can be located correctly.  However,
versions ofsetuptoolsearlier thanv31.0.0does not create the
required package namespace declarations when a package is installed
using this development installation method when mixed withpip installwithin the same namespace.  As a result, inconsistent import
failures can happen for any modules under thecalmjsnamespace.  As an
example:>>>importcalmjs.rjsTraceback(mostrecentcalllast):File"<stdin>",line1,in<module>ImportError:Nomodulenamed'calmjs.rjs'>>>importcalmjs.base>>>importcalmjs.rjs>>>If this behavior (and workaround) is undesirable, please ensure the
installation of allcalmjsrelated packages follow the same method
(i.e. eitherpython setup.py developfor all packages, or using the
wheels acquired throughpip), or upgradesetuptoolsto version
31 or greater and reinstall all affected packages.Testing the installationFinally, to verify for the successful installation ofcalmjs.rjs, the
included tests may be executed through this command:$python-munittestcalmjs.rjs.tests.make_suiteHowever, if the steps to install external Node.js dependencies to the
current directory was followed, the current directory may be specified
as theCALMJS_TEST_ENVenvironment variable.  Under POSIX compatible
shells this may be executed instead from within that directory:$CALMJS_TEST_ENV=.python-munittestcalmjs.rjs.tests.make_suiteDo note that if thecalmjs.devpackage is unavailable, a number of
tests will be skipped.  To avoid this, either install that package
separately, or installcalmjs.rjsusing its extras dependencies
declaration like so:$pipinstallcalmjs.rjs[dev]UsageTo generate a RequireJS artifact from packages that have JavaScript code
exposed through the Calmjs module registry system that are already
installed into the current environment, simply execute the following
command:$calmjsrjsexample.packageThe following sections in this document will provide an overview on how
to enable the JavaScript module export feature for a given Python
package through the Calmjs module registry system, however a more
thorough description on this topic may be found in the README provided
by thecalmjspackage, under the sectionExport JavaScript code from
Python packages.Declaring JavaScript exports for PythonAny exposed JavaScript code through thecalmjs.moduleregistry will
be picked up and compiled into a working RequireJS artifact.  For
example, given the following entry points for that registry defined by a
package namedexample:[calmjs.module]example=exampleThis is the most basic declaration that works for packages that share
the same name as the import location provided.The following is am example for packages that have nested submodules
(calledexample.libandexample.app):[calmjs.module]example.lib=example.libexample.app=example.appWhile the import locations declared looks exactly like a Python module
(as per the rules of a Python entry point), thecalmjs.moduleregistry will present them using the CommonJS/ES6 style import paths
(i.e.'example/lib'and'example/app'), so users of that need
those JavaScript modules to be sure theyrequirethose strings.Please also note that the default source extractor will extract all
JavaScript files within those directories.  Finally, as a consequence of
how the imports are done, it is recommended that no relative imports are
to be used.If the package at hand does not directly declare its dependency oncalmjs, an explicitcalmjs_module_registry=['calmjs.module']may
need to be declared in thesetupfunction for the package to ensure
that this default module registry will be used to acquire the JavaScript
sources from.Putting this together, thesetup.pyfile should contain the
following:setup(name='example',# ... plus other declarations# This is one of the recommended options, even though the# project will not be importing from Calmjs.license='gpl',install_requires=['calmjs>=3.0.0,<4','calmjs.rjs>=2.0.0,<3',# plus other installation requirements],# If the usage of the GPL is impossible for the project, or# declaring a direct dependency on calmjs packages is impossible# for the project for whatever other reasons (even though the# project itself will NOT be required to include/import ANY code# from the calmjs namespace), setup_requires may be used instead# of install_requires, and the following should also be included# as well:package_json={"devDependencies":{"requirejs":"~2.1.17",}},calmjs_module_registry=['calmjs.module'],# the entry points are required to allow calmjs to pick this upentry_points="""
    [calmjs.module]
    example = example
    example.lib = example.lib
    example.app = example.app
    """,)For the construction of the RequireJS artifact, the commandcalmjs rjswill automatically extract all relevant source files from the
dependencies of the selected Python package(s) into a temporary build
directory, where the build manifest will also be generated for the
invocation ofr.jsto construct the artifact.  An example run:$calmjsrjsexampleTracingdependenciesfor:/home/user/example.js/home/user/example.js----------------/tmp/tmp_build/build/example/lib/form.js/tmp/tmp_build/build/example/lib/ui.js/tmp/tmp_build/build/example/lib/main.js/tmp/tmp_build/build/example/app/index.jsAs the build process used bycalmjs.rjsis done in a separate build
directory, all imports through the Node.js module system must be
declared asextras_calmjs.  For instance, ifexample/app/indexneed to use thejqueryandunderscoremodules like so:var$=require('jquery'),_=require('underscore');It will need to declare the target location sourced fromnpmplus the
package_json for the dependencies, it will need to declare this in itssetup.py:setup(# ...package_json={"dependencies":{"jquery":"~3.1.0","underscore":"~1.8.0",},},extras_calmjs={'node_modules':{'jquery':'jquery/dist/jquery.js','underscore':'underscore/underscore.js',},},)Once that is done, rerunpython setup.py egg_infoto write the
freshly declared metadata into the package’s egg-info directory, so that
it can be used from within the environment.calmjs npm--installcan now be invoked to install thenpmdependencies into the current
directory; to permitcalmjs.rjsto find the required files sourced
fromnpmto put into the build directory forr.jsto locate them.The resulting calmjs run may then end up looking something like this:$calmjsrjsexampleTracingdependenciesfor:/home/user/example.js/home/user/example.js----------------/tmp/tmp_build/build/jquery.js/tmp/tmp_build/build/underscore.js/tmp/tmp_build/build/example/lib/form.js/tmp/tmp_build/build/example/lib/ui.js/tmp/tmp_build/build/example/lib/main.js/tmp/tmp_build/build/example/app/index.jsThe transpiler will add the appropriate boilerplates and thus therequirestatements throughrequirejswill import fromnode_modulesif the extras_calmjs have been declared.  However,
there are cases where the desired artifact should only contain the
sources from the Python package without the extras or vice versa (due to
the library being available via another deployed artifact), this is
supported by theempty:scheme byr.js, and to enable it forcalmjs rjsit can be done like so:$calmjsrjsexample--bundlepath-methodempty--export-filenamemain.jsTracingdependenciesfor:/home/user/main.js/home/user/main.js----------------/tmp/tmp_build/build/example/lib/form.js/tmp/tmp_build/build/example/lib/ui.js/tmp/tmp_build/build/example/lib/main.js/tmp/tmp_build/build/example/app/index.js$calmjsrjsexample--sourcepath-methodempty--export-filenamedeps.jsTracingdependenciesfor:/home/user/deps.js/home/user/deps.js----------------/tmp/tmp_build/build/jquery.js/tmp/tmp_build/build/underscore.jsThe above example shows the generation of two separate artifacts, one
containing just the sources from the Python packageexamplethat had
been declared in thecalmjs.moduleregistry, and the other contains
only the external extra sources.If the above triggers a dependency trace error forr.js, there is a
last resort--emptyflag that can be applied; do note that this
completely disables the trace functionality forr.jsas this initiates
a similar trace process to locate all the imported module names for
stubbing them out with theempty:scheme within the generated
configuration file.  Ensure that the modules required by the resulting
artifact has all its required modules provided elsewhere.The explicitextras_calmjsdeclaration also supports the usage
throughbower(supported viacalmjs.bower); instead of usingnode_modulesas the key,bower_componentsshould be used
instead.Alternative registries aside fromcalmjs.modulecan be specified
with the--source-registryflag.  Assuming there are registries in
the current environment registered asmyreg1andmyreg2and theexamplepackage has registered sources to both of them, the command
to build a bundle from both those registries into one artifact can be
triggered like so:$calmjsrjs--source-registry=myreg1,myreg2exampleHandling of RequireJS loader pluginsThe AMD system as defined by RequireJS has the concept of loader
plugins, where the module name provided may be suffixed with a!as
arguments for handling by the plugin.  As the string provided after is
opaque to therequirejssystem as a whole and thus handled directly by
the preceding plugin, the resources that it need will be specific to the
plugin itself.  As it may load resources through therequirejssystem,
any paths that require configuration will need to be done.To account for this issue,calmjs.rjsintroduces the concept of loader
plugin handlers and a registry system for dealing with this.  A givenRJSToolchainwill have a default loader plugin registry assigned,
but this can be overridden by specifying a custom identifier (overriding
the default'calmjs.rjs.loader_plugin') for the registry to be used,
which will allow the handling of very customized loaders for a given
project.  Please refer to thecalmjs.rjs.registrymodule for more
details on how this is constructed and set up for usage.By default, thetexthandler is registered to the default loader
plugin registry, which should cover the most common use case encountered
by thecalmjsframework.  Do note that packages are still required to
declare their (dev)dependencies in theirpackage_jsonto the plugin,
ideally with a well constrained version, so to ensure a consistent build
experience for all end users.Testing standalone, finalized RequireJS artifactsAMD artifacts generated using the standardcalmjs rjstoolchain
runtime may be tested using thecalmjs karmaruntime provided by thecalmjs.devpackage.  Given a finalizedexample.jsthat
implements the features provided by theexamplepackage, the
artifact may be tested with the tests provided by theexamplepackage using the following command:$calmjskarmarun\-tcalmjs.rjs\--artifact=example.js\exampleThe above command invokes the standalone Karma runner using thecalmjs.rjssettings to test against theexample.jsartifact
file, using the tests provided by theexamplepackage.  The test
execution is similar to the one during the development process.Declare prebuilt JavaScript artifacts for Python packagesFinally, to complete the Python package deployment story, the process
should include the automatic generation and inclusion of the JavaScript
artifacts in the resulting Python wheel.  This can be achieved by
specifying an entry in thecalmjs.artifactsregistry, with the key
being the filename of the artifact and the value being the import
location to a builder.  A default builder function provided atcalmjs.rjs.artifact:complete_rjswill enable the generation of a
complete RequireJS artifact for the Python package.  For example:[calmjs.artifacts]example.package.rjs.js=calmjs.rjs.artifact:complete_rjsOnce those entry points are installed, runningcalmjs artifact build example.packagewill make use of the RequireJS toolchain and build the
artifact atexample.package.rjs.jsinside thecalmjs_artifactsdirectory within the metadata directory forexample.package.
Alternatively, for solution more integrated withsetuptools, thesetupfunction insetup.pyshould also enable thebuild_calmjs_artifactsflag such thatsetup.py buildwill also
trigger the building process.  This is useful for automatically
generating and including the artifact as part of the wheel building
process.  Consider thissetup.py:setup(name='example.package',# ... other required fields truncatedbuild_calmjs_artifacts=True,entry_points="""
    # ... other entry points truncated
    [calmjs.module]
    example.package = example.package

    [calmjs.artifacts]
    example.package.rjs.js = calmjs.rjs.artifact:complete_rjs
    """,)Building the wheel usingsetup.pymay result in something like this.
Note that the execution ofr.jswas part of the process and that the
metadata (egg-info) directory was then built into the wheel.$ python setup.py bdist_wheel
running bdist_wheel
running build
...
running build_calmjs_artifacts
automatically picked registries ['calmjs.module'] for sourcepaths
using loaderplugin registry 'calmjs.rjs.loader_plugin'
...
/src/example.package.egg-info/calmjs_artifacts/example.package.rjs.js
----------------
/tmp/tmpm_2jf151/build/example/package/index.js
...
running install_egg_info
Copying src/example.package.egg-info to build/.../wheel/example.package...
running install_scripts
creating build/.../wheel/example.package-1.0.dist-info/WHEELFor testing the package artifact, the following entry point should also
be specified under thecalmjs.artifacts.testsregistry, such that
runningcalmjs artifact karma example.packagewill execute the
JavaScript tests declared byexample.packageagainst the artifacts
that were declared incalmjs.artifacts.[calmjs.artifacts.tests]example.package.rjs.js=calmjs.rjs.artifact:test_complete_rjsTroubleshootingThe following are some known issues with regards to this package and its
integration with other Python/Node.js packages.When callingcalmjs rjson a package, gotENOENTTypically this is caused by source files from the source package or
registry invokingrequirea JavaScript module that is not available
in the build directory.  One possible cause through thecalmjsframework is that the Python package failed to declareextras_calmjsthat it might require, or that explicit map method and/or source
registry that was selected did not result in all required sources be
loaded into the build directory.If the missing source files are intended, applying the--emptyor
the-eflag to therjstool will stub out all the missing
modules from the bundle; do note that this will result in the generated
artifact bundle not having all the required modules for its execution.
The resulting artifact bundle should be used in conjunction with the
other artifact bundles that provide the result of the required
dependencies.RJSRuntimeError: unable to locate ‘r.js’This means the current Node.js environment is missing the requirejs
package fromnpm; either install it manually with it or throughcalmjson this package.  If a given Python package is required to use
requirejs to generate the package, itspackage_jsonshould declare
that, or declare dependency oncalmjs.rjs.UserWarning: Unknown distribution option:During setup and installation using the development method, if this
warning message is shown, please ensure the egg metadata is correctly
generated by runningpython setup.py egg_infoin the source
directory, as the packagecalmjswas not available when the setup
script was initially executed.ContributeIssue Tracker:https://github.com/calmjs/calmjs.rjs/issuesSource Code:https://github.com/calmjs/calmjs.rjsLegalThecalmjs.rjspackage is part of the calmjs project.The calmjs project is copyright (c) 2016 Auckland Bioengineering
Institute, University of Auckland.calmjs.rjsis licensed under the
terms of the GPLv2 or later.Changelog2.0.1 (2018-05-03)Update the export_target production and usage of working_dir to be
inline with what is expected bycalmjs-3.1.0. [#3]2.0.0 (2018-01-12)Support forcalmjs-3.0.0features and breaking changes.Loader plugin framework migrated upstream; downstream packages that
make use of them should no longer declare explicit entries inextras_calmjsto permit wider portability.Removed usage ofslimitin favor of the capabilities now provided
bycalmjsandcalmjs.parse.The flags for thecalmjs rjsruntime have been changed to remove
some naming confusion, mainly due to sourcemap and also to maintain
consistency with othercalmjstools.--bundle-map-methodis deprecated in favor for--bundlepath-method; will be fully removed by 3.0.0--source-map-methodis deprecated in favor for--sourcepath-method; will be fully removed by 3.0.0Provide a generic package-level artifact builder for thecalmjs.artifactsregistry along with the respective tester for thecalmjs.artifacts.testsregistry.1.0.2 (2017-05-22)Corrected the issue where plugins that have been unmapped using theempty:scheme triggeringFileNotFoundError.1.0.1 (2017-01-27)Load the non-test files in deps also, instead as part of the tests to
avoid automatic inclusion.Test files should start with the name test as per convention.1.0.0 (2016-11-18)Initial implementation that brings in the support of the production of
AMD artifacts (bundles) from JavaScript sources included with Python
packages (along with their declared dependencies throughnpmor
other supported tools) through the calmjs framework.Enabled thecalmjs rjstool entry point.Also provide integration withcalmjs.devby correcting the correct
hooks so that this package can be used as an advice package for the
execution of tests against artifacts generated through this package.