Package: cels
celsCommand line tool to patch your YAML and JSON filesExample# input.yamlfoo:bar:1baz:2list:-a-blevel:11# patch.yamlfoo:bar:100list {insert}:clevel {delete}:null# command: cels patch input.yaml patch.yamlfoo:bar:100baz:2list:-a-b-cDescriptionCels is a command-line tool and Python library that enables you to make multiple
modifications to YAML, JSON, or TOML documents. These modifications are based on
changes specified in apatchfile. The patch file is written in the same
format as the original data and mirrors its structure. For instance, if you want
to change thebarkey value fromhellotobyein this example:foo:bar:helloYou simply need to create a patch like this:foo:bar:byeTo run the tool and get the result, you just pass both files as arguments to
thecels patchcommand:$celspatchinput.yamloutput.yamlFor more complex modifications, you can annotate the keys of the patch
document with the operation to perform in the format:{operation[@index1, index2, …]}. For example, to insert an element in the middle of a list, you
only need to specify the new location and the value to insert:# inputlist:-a-c# patchlist {insert@1}:b# outputlist:-a-b-cWorking with JSON files in Cels is quite similar to working with YAML files. To
illustrate this, let's take a look at how the previous example would be
transformed:Here's the input file:{"list":["a","c"]}And here's the patch file:{"list {insert@1}":"b"}After applying the patch, the result would be:{"list":["a","b","c"]}Cels supports a variety of operations (set,delete,rename,insert,extend,use,link,render), and most of them can optionally take indices
to work with any level of nested lists.Cells can be beneficial when you possess a base configuration file or manifest
for a system and need to tailor it to various environments (eg. 'development'
and 'production'). Additionally, you can utilize Cells as a Python library and
integrate it into your application. This allows your users to modify their own
configuration in diverse setups (similar todocker-composeoverrides, for
example).Refer toUsagefor a comprehensive description of all available
operations.SeeSimilar Projectsfor a comparison between Cels and
other tools and specifications with similar objectives.The project nameThe name 'Cels' is inspired by the conventional, analog world of animation. In
that context, 'cels' are transparent sheets featuring drawings that are layered
atop one another to create the final image. Cels works in a similar way,
creating the final result by layering the patch files over the input documents.LimitationsIf you're considering using Cels, please take into account the following limitations:Cels is only compatible with documents where the top-level element is
a dictionary or map. That is, won't work with documents where the top-level
element is a list.Annotations can only be added to keys that are of the string type.
This isn't a problem with JSON documents, as all keys are inherently strings.
However, in YAML documents, keys aren't necessarily strings. While Cels can
handle YAML documents with non-string keys—and these keys can also appear in
patch files—they cannot be annotated. This means that advanced operations
cannot be applied to them.InstallationTo install cells, simply usepip:$pipinstallcellsUsageThe 'patch' commandCels' main command ispatch. It takes an input file (the file that you want to
modify) and a patch file (the file that describes the changes to perform). Eg:$celspatchinput.yamlpatch.yamlThe result or running the command is sent tostdoutby default. To save the
result to a file, just redirect the output or use the-Ooption:$celspatchinput.yamlpatch.yaml>output.yaml
$celspatchinput.yamlpatch.yaml-Ooutput.yamlYou can patch YAML, JSON and TOML files. Cels determines the correct format to
use from the file extensions, but you can explicitly set the format for each one
of the files with the-i,-p, and-oparameters (runcels patch --helpfor more information).Note: It's possible to utilize various formats within the same command
execution. For instance, you can use a JSON patch and a YAML input file
simultaneously if you wish. Although there aren't many reasons to typically do
this, Cels doesn't place any limitations on the combination of formats used.Basic operationsTo override values of the input document, you don't typically need to specify
any operation in the patch file:# inputfoo:bar:a:1b:2c:3# patchfoo:bar:b:200# outputfoo:bar:a:1b:200c:3As shown in this example, dictionaries in the patch file get merged with
the ones in the input file. However, if you want to override an entire
dictionary, instead of merging it, you can use the operationset:# inputfoo:bar:a:1b:2c:3# patchfoo:bar {set}:b:200# outputfoo:bar:b:200Operations—in their most basic form—are specified by appending the operation
name between curly braces ({<operation>}) to the key string. The full
annotation format is:key {<operation>[@<index1>,<index2>,...]}where the indices are provided to modify lists (see belowWorking with
lists).'delete' and 'rename'The other basic operations includedelete, which removes a key, andrename,
which alters the key's name while preserving its value.# inputfoo:bar:"AFairFieldFullofFolk"baz:one:1two:2# patchfoo:bar {rename}:newbarbaz {delete}:null# outputfoo:newbar:"AFairFieldFullofFolk"For thedeleteoperation, you can pass any value (in this casenull) as it
is just ignored by Cels.Working with ListsThe majority of operations can utilize an index following an@symbol. This
is to indicate that the action should be executed at a specific position within
a list (To know if a given operation supports indices, runpatch describe operation OPERATION_NAME, which will provide full information about it).For example, to change the second element of a list, you can use thesetoperation:# inputfoo:-a-b# patchfoo {set@1}:B# outputfoo:-a-BNote that when working with lists:Indices start at 0 (meaning the first element is located at index 0).Negative indices can be used. For instance, -1 refers to the last element in
the list, -2 to the penultimate one, and so on.To specify elements within nested lists, you can provide multiple indices
separated by commas. For example,1,0refers tocin a nested list like[[a, b], [c, d]].'insert' and 'extend'The operationsinsertandextendare very useful when working with lists.insertadds one element to a list:# inputfoo:-a-b# patchfoo {insert}:c# outputfoo:-a-b-cWhereas, in the case ofextend, multiple elements are added simultaneously:# inputfoo:-a-b# patchfoo {extend}:-c-d# outputfoo:-a-b-c-dIn both scenarios, indices can be used to designate the position where you wish
to add elements:{insert@NUM}and{extend@NUM}place the elementsbeforetheNUMposition. In other words, to insert them at the start of the list, before
any other element, use{insert@0}and{extend@0}.The unique index_can be utilized to signify that the elements should beappendedto the list's end (i.e.,{insert@_}). The_can be omitted when
operating on the top-level list, meaning{insert}and{extend}are
synonymous with{insert@_}and{extend@_}. However, if you need to append
elements to a nested list's end, it's necessary to use it.For example, here's how you append an element to the end of a nested list:# inputfoo:--a-b--c-d# patchfoo {insert@1,_}:e# output--a-b--c-d-eUsing variablesIt is possible to define variables and then reuse them at different places of
the patch file.This is a quick example:# inputfoo:1# patchmy_var {var}:Worldfoo {use}:my_varbar {render}:"Hello{{my_var}}!"# outputfoo:Worldbar:"HelloWold!"As you can see, variables are defined with:key {var}:valuewherekeyis the name of the variable andvalueis, well, its value.Variable definitions, while not visible in the output document themselves, can
be utilized through theuseandrenderoperations to insert values in
various locations:Theuseoperation simply inserts the variable as is. If the variable is a
list or a dictionary,usewill incorporate it without any modifications.Therenderoperation allows you to define aJinjatemplate
string that can reference one or more variables. If the variable is a list or
a dictionary, you can use the.or[]notation to pinpoint the exact
value you wish to use. Additionally, you can utilize any of the features
offered by the Jinja template language, such as filters or conditional
structures.The following is a more elaborated example of using variables with therenderoperation:# patchmy_var {var}:one:atwo:-b-cfoo {render}:"{{my_var.one}}lowercase,{{my_var.two[0]|upper}}uppercase"# outputfoo:a lowercase, B uppercaseVariable scopePlease note that a variable comes with an associated scope:Theuseandrendercommands can only reference variables from the same
dictionary in which they are used, or from any parent or ancestor dictionary.
In other words, if you want a variable to be accessible throughout the entire
document, you should define it at its root dictionary.If a variable is redefined in a child dictionary, the value in the child
dictionary will take precedence over the one in the parent dictionary.Multiple changes for a same keyIf you want to perform multiple changes to the same key, you can use thechangeoperation, that takes a list of the modifications to perform:# inputfoo:1# patchfoo {change}:-operation:setvalue:100-operation:renamevalue:bar# outputbar:100The operations will be executed in the order they are presented. Each item in
the operations list should contain no more than three fields:operation: This refers to one of the possible operations (set,delete,rename,insert, etc.).value: The value depends on the type of operation. It may be omitted for
keys that do not require a value, such asrename.indices: This is a list of integers (and_forinsertandextendoperations) used to manipulate nested lists. This field can be omitted for
operations that do not intend to modify lists.In summary, these three fields correspond to those that can be defined in a
standard annotation:key {operation@indices}: value.Note: the notation:foo {change}:
  - operation: set
    value: 100
  - operation: rename
    value: baris different from listing the operations one after another:foo {set}: 100
foo {rename}: barIn the latter scenario, the second operation would supersede the initial one.
Therefore, there are minimal instances, if any, where you might want to do
that.Repeating content from the input documentThelinkoperation allows you to reference parts of the input document and
reuse them in other parts. For instance, here we use thelinkoperation to
remove thebarlevel from the input document:# inputfoo:bar:one:1two:2# patchfoo {delete}:nullnew-foo {link}:.foo.bar# resultnew-foo:one:1two:2The 'link' operation always takes a path that employs the '.' and '[]' notation
to traverse through the dictionaries and lists within the input document. The
initial '.' symbol signifies the root dictionary of the document. It's important
to note that the path always refers to the unaltered input document, regardless
of any other operations being performed. This means that the value indicated by
the path will always be the original one.Patching dictionaries that are nested in listsIf you need to modify a dictionary within a list, you can utilize thepatchoperation. In most scenarios, explicitly usingpatchisn't required, as
dictionaries in the patch file are automatically merged with those in the input
document. However, when dealing with dictionaries within lists, it becomes
essential to usepatchbecause you need to specify the index that indicates
the dictionary's position within the list. Here's an example:# inputfoo:-a:1b:2# patchfoo {patch@0}:a:100# resultfoo:-a:100b:2Changing the annotation formatBy default, annotations in Cels appear as<space>{operation@indices}. However,
you can customize all the symbols used to better suit your needs by using the
following command parameters:$celspatchinput.yamlpatch.yaml\--separator"_"\--left-marker"("\--index-marker"%"\--right-marker")"This example changes the format of annotations to:_(operation%indices).Getting helpTo list all available operations, you can use:$celslistoperationsTo show help for a given operation, including its description and usage
examples, you can use:$celsdescribeoperationOPERATION_NAMEAdditionally, with the-vflag, you can activate the verbose output.
This will display the operation that was used to generate each node in the
output document:Using Cels as a Python libraryYou can use Cels programmatically from your Python code. It provides two basic
functions:patch_documentandpatch_dictionary.patch_documentpatch_documentallows you to pass JSON, YAML or TOML text for the input and
patch documents, in the same way that they are passed to thecelscommand:fromcelsimportpatch_documentoutput=patch_document(input_format,input_text,patch_format,patch_text,output_format)input_formatandpatch_formatare string arguments and
can take any of the following values:json,yamlortoml.input_textandpatch_textare the raw texts to be used (of course, their
format should match with the parameters above).Finally, it is possible to specify the format of the output text withoutput_format, which doesn't necessarily have to match the input formats.patch_dictionarypatch_dictionaryworks exactly the same aspatch_documentbut the data is
passed already in the format of python dictionaries:fromcelsimportpatch_dictionaryresult=patch_dictionary(input_dict,patch_dict)In both cases (patch_documentandpatch_dictionary), you can passseparator,left_marker,index_markerandright_markerparameters to
define the format of the key annotations (seeChanging the annotation
formatfor more information).DeepcopyFor performance reasons, Cels' Python functions do not generate a complete copy of the
output dictionary in memory. Instead, Cels interlaces the input and patch
nodes, merging them to produce the final result.This approach works well if you only need to read the output. However, if you
alter it in-place, you may inadvertently modify nodes of the input dictionary
at the same time.If you need to change the output dictionary without affecting the input and
patch dictionaries, simply create adeep copyof the dictionary returned by
the Cels functions:fromcopyimportdeepcopyfromcelsimportpatch_dictionaryresult=deepcopy(patch_dictionary(input_dict,patch_dict))Similar Projectsjq and yqjqandyqare widely used command-line tools for processing JSON and YAML. The key
distinction between these tools and Cels lies in their operational approach.jqandyqoperate using paths, whereas Cels utilizes patch files.For example, if you want to alter the value of a key inyq, you would specify
it as follows:.foo.bar.baz = "value"In contrast, with Cels, you would write the actual YAML:foo:
  bar:
    baz: "value"jqandyqshine when it comes to making specific modifications to a
document. They allow you to pinpoint and alter a deeply nested key with a
single command line. However, their intuitiveness diminishes when multiple
changes are required within a document. In such scenarios, using a patch file
as in Cels may provide a better overall view of the modifications and their
interrelations.In this discussion, we're primarily focusing on the patching capabilities of
these tools. However, they also offer additional features (like formatting and
data extraction) that Cels does not.
For more details, please refer to their respective documentation.It's worth noting thatjqis exclusively for JSON, whileyqcan handle both
JSON and YAML.Jsonnet, CUE, YTTJsonnet,CUE, andYTTare fully-fledged languages equipped with import
mechanisms, loops, conditionals, functions, and other programming language
constructs. They are designed as supersets of JSON, as seen in Jsonnet and CUE,
and a superset of YAML in the case of YTT (however, in this case, language
constructs are embedded within comments, ensuring compatibility as a YAML
document). These languages can be an excellent choice if you're tasked with
making complex modifications to JSON or YAML files.Cels stands out from these solutions due to its simplicity. All you need to do
is annotate each key you wish to modify with the desired operation. While it
may not have the capabilities of Jsonnet, CUE, YTT, or any other configuration
programming language, it still covers a wide range of common use cases while
still being extremely simple to use.Starlark and DhallStarlarkandDhallshare similarities with Jsonnet, CUE, and YTT
as they are all fully-fledged configuration programming languages. However,
they don't extend JSON or YAML. Starlark is a subset of Python, while Dhall has
its own unique syntax (though it does allow exporting to any other format).As in the case of, Jsonnet, CUE, and YTT, Cels sets itself apart from Starlark
and Dhall by beingnota comprehensive language, but rather a simple
collection of annotations. This makes it considerably less complex in
comparison.RFC 6902 and RFC 7396RFC 6902andRFC
7396are both proposed
standards for patching JSON files.RFC 6902defines a JSON structure for defining a list of operations to
be applied sequentially to the original document in order to patch it.This is an example:[
  { "op": "remove", "path": "/a/b/c" },
  { "op": "add", "path": "/a/b/c", "value": [ "foo", "bar" ] },
  { "op": "replace", "path": "/a/b/c", "value": 42 },
]RFC 6902provides a broad spectrum of operations. However, it doesn't
replicate the original document, but merely outlines the operations in a list
format. On the other hand, the patch file in Cels can result in a more compact
document that is easier to read and potentially easier to maintain. Moreover,
Cels isn't limited to JSON support; it also accommodates YAML and TOML.On the other hand,RFC 7396is very similar to Cels. Like Cels, it defines a
patch format that mirrors the original document.For example, given the following example file:{
  "a": "b",
  "c": {
    "d": "e",
    "f": "g"
  }
}A RFC 7396 patch may look like this:{
  "a":"z",
  "c": {
    "f": null
  }
}Which results in:{
  "a": "z",
  "c": {
    "d": "e",
  }
}(Setting a key tonulldeletes it).For comparison, the equivalent Cels patch would look like:{
  "a":"z",
  "c": {
    "f {remove}": null
  }
}As shown in the examples,RFC 7396and Cels are very similar conceptually,
however:RFC 7396only supports leaving a key untouched, overriding it or deleting
it. Cels, on the other hand, offers a wider range of options.Cels can operate on the elements of lists, while that's not possible inRFC 7396.RFC 7396only supports JSON, while Cels can work with YAML and TOML too.RFC 7396usesnullto delete keys from the original dictionary. However,nullis a completely valid value in a JSON file, which makesRFC 7396unable of representing certain valid JSON documents (in particular, it is not
possible to set a key tonullas it would be deleted instead).These examples illustrate thatRFC 7396and Cels share many similarities, but
there are key differences:RFC 7396is limited to leaving a key as is, overwriting it, or deleting it.
In contrast, Cels provides a broader set of options.Cels has the capability to manipulate list elements, a feature thatRFC 7396lacks.RFC 7396employsnullto remove keys from the original dictionary.
However,nullis a perfectly valid value in a JSON file, which rendersRFC 7396incapable of representing certain valid JSON documents.
Specifically, it's impossible to assign anullvalue to a key as it
would be deleted instead.WhileRFC 7396only supports JSON, Cels can handle YAML and TOML as well.The following is a (non-comprehensive) list ofRFC 6902andRFC7396implementations:json-patch: Go implementation of RFC 6902 and
RFC 7396.python-json-patch: Python
implementation of RFC 6902.yaml-diff-patch: Command line
and npm package that allows to apply RFC 6902 JSON patches to a YAML document.chbrown/rfc6902: TypeScript
implementation of RFC 6902.LicenseCels is available under the MIT license.