Package: xtce-generator
xtce_generatorA tool to generate xtce files from a sqlite database. This database is assumed to have been generated byjucier.
We follow the XTCE specifications described onXTCE Verison 1.2as per OMG document number:formal/18-10-04. You may find this documenthere.XTCE Patterns, Conventions and CaveatsNotes For DevelopersDependenciessix==1.15.0PyYAML~=5.3.1setuptools~=50.3.2flake8twineNOTE:If you have issues running venv for python3.6 on Ubuntu 16, you should be able to fix it with this:sudo apt install python3.6-venv python3.6-devHow to useBe sure to run this from a virtual environment:python xtce_generator.py  --sqlite_path /home/vagrant/auto-yamcs/juicer/newdb --log_level ['DEBUG', 'INFO', `WARNING`, `ERROR`, `CRITICAL`, `SILENT`] --config_yaml /home/vagrant/xtce_generator/src/xtce_generator/config.yaml  --spacesystem ocpocAfter xtce_generator is done, you should have a file with the name of space_system_name.xml. In our case this isocpoc.xmlsince we passedocpocas our root spacesystem.
Please beware that you may pass in any name for your spacesystem; this depends on your working context. It could be a vehicle name, machine architecture or the name of a space mission itself!Using The XMLThe auto-generated xtce file,ocpoc.xmlin our case, is ready for use for any ground system that is xtce-compliant. For now we useyamcs. Here is an example demonstrating such use:Copy your xtce to your yamcs project:cp ~/xtce_generator/src/ocpoc.xml /home/vagrant/yamcs-cfs/src/main/yamcs/mdbBe sure to add this file to youryamcsinstance file:...
mdb:
    # Configuration of the active loaders
    # Valid loaders are: sheet, xtce or fully qualified name of the class
    - type: "xtce"
      spec: "mdb/cfs-ccsds.xml"
    - type: "xtce"
      spec: "mdb/CFE_EVS.xml"
    - type: "xtce"
      spec: "mdb/ocpoc.xml"Then all you do is run yamcs!mvn yamcs:runXTCE ComplianceIn the future we will strive to be 100% xtce compliant. However, given that we useyamcsas our ground system, if there are quirks yamcs has we will be adhering to those quirks. For more info on this, you may gohere.XTCE  Data StructuresOur xtce data structures are generated withgenerateDS 2.36.2. There is a quirk with this tool where it will be nameSpaceSystemobjects toSpaceSystemTypeafter generating the python code the xtce schema. This is very easy to fix.Go to theexportmethod of yourSpaceSystemType.
Change it fromdef export(self, outfile, level, namespaceprefix_='', namespacedef_='', name_='SpaceSystemType', pretty_print=True):todef export(self, outfile, level, namespaceprefix_='', namespacedef_='', name_='SpaceSystem', pretty_print=True):After renaming that parameter, you are good to go!
**NOTE: **Beware that if you do not change this,yamcswillnotrun.XTCE Patterns and ConventionsWhile we strive for 100% XTCE compliance, we lean towards YAMCS's version of XTCE, which isalmostfully XTCE compliant. Below are the specifications of the patterns, naming conventions and overall structure we follow  to generate our XTCE files.NamespacesAs the XTCE standard mandates, our namespaces are enclosed with the<xtce:SpaceSystem>tag. More specfically, we have aBaseTypenamespace. This namespace hasallof the necessary base types that a SpaceSystem may use. Please note that we use the termsNamespaceandSpaceSysteminterchangeably. A base type is something that is notusuallyuser-defined, but is rather machine-defined. Below is a table with examples of base types that may be part of ourBaseTypenamespace.NameDescriptionSize in bitsEncodingEndiannessuint1_LEAn unsigned integer.1unsignedLittle Endianuint16_BEAn unsigned integer.16unsignedBig endianchar160_LEA string with 160 bits, or 20 characters.160UTF-8Little EndianOur naming conventions for base types willalwaysbe [type][number_of_bits][under_score][Endianness] just lke you see in the table above.Please note that these base types are nearly identical to what you'd find in a programming language like C. That is because that is what they are meant to represent. Our goal is to auto-generate our XTCE files by extracting data(structs, variables, etc) from binary and executable files. Our reasons for this approach are simple; writing XTCE files by hand is error-prone and time consuming. At the moment the tool chain that will allow us to do this isalmostdone. One of those tools isjuicer, which extracts DWARF data from ELF files, such as executable binary files.DefaultRateInStream<SequenceContainer name="HES_HK_TLM_MID">
    <xtce:DefaultRateInStream basis="perSecond"
        xsi:type="xtce:RateInStreamType"
        minimumValue="250" />
    <EntryList>
        <ParameterRefEntry parameterRef="HES_HK_TLM_MID"/>
    </EntryList>
    <BaseContainer containerRef="cfs/cfs-tlm-hdr">
        <RestrictionCriteria>
            <ComparisonList>
                <Comparison parameterRef="cfs/ccsds-apid" value="865"/>
            </ComparisonList>
        </RestrictionCriteria>
    </BaseContainer>
</SequenceContainer>This xtce xml node allows users to specify the min/max rate a container packet is supposed to be received by the ground station.
At the moment onlyminimumValueis supported as that is what YAMCS supports(Yamcs 5.7.9, build c654828f93607440ffee33c5bed72a98af775cd8) at the time of writing.ParameterTypesOur parameter types arealwaysdefined under<TelemetryMetaData>. The naming of our parameter types follows the convention shown in the table above.ArgumentTypesOur argument types, as per XTCE design and standard, arealwaysdefined under<CommandMetaData>. The naming of our arguments types follows the convention shown in the table above.User-defined TypesEnumeratedParameterType,AggregateParamterType,AggregateArgumentTypeandEnumeratedArgumentTypetypes are what we consider user-defined types. This means that these types arenotpart of theBaseTypenamespace.EnumeratedParameterTypeandAggregateParamterTypetypes arealwaysdefined on<TelemetryMetaData>.AggregateArgumentTypeandEnumeratedArgumentTypearealwaysdefined on<CommandMetaData>.AggregateArgumentTypemembers are flattened. This won't affect much of anything since the outcome is the same as if they were bundled inside an AAggregate.ArraysArrays Inside Structures(*AggregateType)Standalone( that is it does not appear inside a structure) arrays arenotsupportedThe following is the example of a structure(AggregateParamterType) using an array:<AggregateParameterType name="CFE_ES_MemPoolStats_t">
                            <MemberList>
                                <Member name="PoolSize" typeRef="BaseType/uint32_LE"/>
                                <Member name="NumBlocksRequested" typeRef="BaseType/uint32_LE"/>
                                <Member name="CheckErrCtr" typeRef="BaseType/uint32_LE"/>
                                <Member name="NumFreeBytes" typeRef="BaseType/uint32_LE"/>
                                <Member name="BlockStats" typeRef="BaseType/Array_16Dim__simlink_core_cfe_cfe_es_CFE_ES_BlockStats_t"/>
                            </MemberList>
                        </AggregateParameterType>Here we have a structure calledCFE_ES_MemPoolStats_t(notice the naming convention explained above). It has an array called BlockStats which points to an array in BaseType namespace.The way arrays are constructed was heavily inspired byDWARF4andXTCEstandards.Base types(Intrinsic types)The types shown above such as uint16_LE are what is known as a base type. These are the types that represent things like int, float, char, etc in code. As mentioned above thesewillalways be part of theBaseTypenamespace. Below is a table with all of our base types. Please note that this table is subject as we have not fully standarized our xtce format yet.
The endianness cells marked with a "*" at the end areexclusivelydescribingjustbit ordering and NOT byte-ordering.
As you see on the table this just applies to the types that are 8 bits or less as byte-ordering is irrelevant for these.
We considerbitordering for all of our types for the rare case where there is an architecture that takes into count bit ordering.NameDescriptionSize in bitsEncodingEndiannessuint1_BEAn unsigned integer.1unsignedBig Endian*uint1_LEAn unsigned integer.1unsignedBig Endian*uint2_BEAn unsigned integer.2unsignedBig Endian*uint2_LEAn unsigned integer.2unsignedLittle Endian*int2_BEA signed integer.2signedBig Endian*int2_LEA signed integer.2signedLittle Endian*uint3_BEAn unsigned integer.3unsignedBig Endian*uint3_LEAn unsigned integer.3unsignedLittle Endian*int3_BEA signed integer.3signedBig Endian*int3_LEA signed integer.3signedLittle Endian*uint4_BEAn unsigned integer.4unsignedBig Endian*uint4_LEAn unsigned integer.4unsignedLittle Endian*int4_BEA signed integer.4signedBig Endian*int4_LEA signed integer.4signedLittle Endian*uint5_BEAn unsigned integer.5unsignedBig Endian*uint5_LEAn unsigned integer.5unsignedLittle Endian*int5_BEA signed integer.5signedBig Endian*int5_LEA signed integer.5signedLittle Endian*uint6_BEAn unsigned integer.6unsignedBig Endian*uint6_LEAn unsigned integer.6unsignedLittle Endian*int6_BEA signed integer.6signedBig Endian*int6_LEA signed integer.6signedLittle Endian*uint7_BEAn unsigned integer.7unsignedBig Endian*uint7_LEAn unsigned integer.7unsignedLittle Endian*int7_BEA signed integer.7signedBig Endian*int7_LEA signed integer.7signedLittle Endian*uint8_BEAn unsigned integer.8unsignedBig Endian*uint8_LEAn unsigned integer.8unsignedLittle Endian*int8_BEA signed integer.8signedBig Endian*int8_LEA signed integer.8signedLittle Endian*uint9_BEAn unsigned integer.9unsignedBig Endianuint9_LEAn unsigned integer.9unsignedLittle Endianint9_BEA signed integer.9signedBig Endianint9_LEA signed integer.9signedLittle Endianuint10_BEAn unsigned integer.10unsignedBig Endianuint10_LEAn unsigned integer.10unsignedLittle Endianint10_BEA signed integer.10signedBig Endianint10_LEA signed integer.10signedLittle Endianuint11_BEAn unsigned integer.11unsignedBig Endianuint11_LEAn unsigned integer.11unsignedLittle Endianint11_BEA signed integer.11signedBig Endianint11_LEA signed integer.11signedLittle Endianuint12_BEAn unsigned integer.12unsignedBig Endianuint12_LEAn unsigned integer.12unsignedLittle Endianint12_BEA signed integer.12signedBig Endianint12_LEA signed integer.12signedLittle Endianuin13_BEAn unsigned integer.13unsignedBig Endianuint13_LEAn unsigned integer.13unsignedLittle Endianint13_BEA signed integer.13signedBig Endianint13_LEA signed integer.13signedLittle Endianuint14_BEAn unsigned integer.14unsignedBig Endianuint14_LEAn unsigned integer.14unsignedLittle Endianint14_BEA signed integer.14signedBig Endianint14_LEA signed integer.14signedLittle Endianuint15_BEAn unsigned integer.15unsignedBig Endianuint15_LEAn unsigned integer.15unsignedLittle Endianint15_BEA signed integer.15signedBig Endianint15_LEA signed integer.15signedLittle Endianuint16_BEAn unsigned integer.16unsignedBig Endianuint16_LEAn unsigned integer.16unsignedLittle Endianint16_BEA signed integer.16signedBig Endianint16_LEA signed integer.16signedLittle Endianuint17_BEAn unsigned integer.17unsignedBig Endianuint17_LEAn unsigned integer.17unsignedLittle Endianint17_BEA signed integer.17signedBig Endianint17_LEA signed integer.17signedLittle Endianuint18_BEAn unsigned integer.18unsignedBig Endianuint18_LEAn unsigned integer.18unsignedLittle Endianint18_BEA signed integer.18signedBig Endianint18_LEA signed integer.18signedLittle Endianuint19_BEAn unsigned integer.19unsignedBig Endianuint19_LEAn unsigned integer.19unsignedLittle Endianint19_BEA signed integer.19signedBig Endianint19_LEA signed integer.19signedLittle Endianuint20_BEAn unsigned integer.20unsignedBig Endianuint20_LEAn unsigned integer.20unsignedLittle Endianint20_BEA signed integer.20signedBig Endianint20_LEA signed integer.20signedLittle Endianuint21_BEAn unsigned integer.21unsignedBig Endianuint21_LEAn unsigned integer.21unsignedLittle Endianint21_BEA signed integer.21signedBig Endianint21_LEA signed integer.21signedLittle Endianuint22_BEAn unsigned integer.22unsignedBig Endianuint22_LEAn unsigned integer.22unsignedLittle Endianint22_BEA signed integer.22signedBig Endianint22_LEA signed integer.22signedLittle Endianuint23_BEAn unsigned integer.23unsignedBig Endianuint23_LEAn unsigned integer.23unsignedLittle Endianint23_BEA signed integer.23signedBig Endianint23_LEA signed integer.23signedLittle Endianuint24_BEAn unsigned integer.24unsignedBig Endianuint24_LEAn unsigned integer.24unsignedLittle Endianint24_BEA signed integer.24signedBig Endianint24_LEA signed integer.24signedLittle Endianuint25_BEAn unsigned integer.25unsignedBig Endianuint25_LEAn unsigned integer.25unsignedLittle Endianint25_BEA signed integer.25signedBig Endianint25_LEA signed integer.25signedLittle Endianuint26_BEAn unsigned integer.26unsignedBig Endianuint26_LEAn unsigned integer.26unsignedLittle Endianint26_BEA signed integer.26signedBig Endianint26_LEA signed integer.26signedLittle Endianuint27_BEAn unsigned integer.27unsignedBig Endianuint27_LEAn unsigned integer.27unsignedLittle Endianint27_BEA signed integer.27signedBig Endianint27_LEA signed integer.27signedLittle Endianuint28_BEAn unsigned integer.28unsignedBig Endianuint28_LEAn unsigned integer.28unsignedLittle Endianint28_BEA signed integer.28signedBig Endianint28_LEA signed integer.28signedLittle Endianuint29_BEAn unsigned integer.29unsignedBig Endianuint29_LEAn unsigned integer.29unsignedLittle Endianint29_BEA signed integer.29signedBig Endianint29_LEA signed integer.29signedLittle Endianuint30_BEAn unsigned integer.30unsignedBig Endianuint30_LEAn unsigned integer.30unsignedLittle Endianint30_BEA signed integer.30signedBig Endianint30_LEA signed integer.30signedLittle Endianuint31_BEAn unsigned integer.31unsignedBig Endianuint31_LEAn unsigned integer.31unsignedLittle Endianint31_BEA signed integer.31signedBig Endianint31_LEA signed integer.31signedLittle Endianuint32_BEAn unsigned integer.32unsignedBig Endianuint32_LEAn unsigned integer.32unsignedLittle Endianint32_BEA signed integer.32signedBig Endianint32_LEA signed integer.32signedLittle Endianuint33_BEAn unsigned integer.33unsignedBig Endianuint33_LEAn unsigned integer.33unsignedLittle Endianint33_BEA signed integer.33signedBig Endianint33_LEA signed integer.33signedLittle Endianuint34_BEAn unsigned integer.34unsignedBig Endianuint34_LEAn unsigned integer.34unsignedLittle Endianint34_BEA signed integer.34signedBig Endianint34_LEA signed integer.34signedLittle Endianuint34_BEAn unsigned integer.34unsignedBig Endianuint34_LEAn unsigned integer.34unsignedLittle Endianint35_BEA signed integer.35signedBig Endianint35_LEA signed integer.35uint_BEAn unsigned integer.uint35_LEAn unsigned integer.35unsignedLittle Endianint35_BEA signed integer.35signedBig Endianuint36_BEAn unsigned integer.36unsignedBig Endianuint36_LEAn unsigned integer.36unsignedLittle Endianint36_BEA signed integer.36signedBig Endianint36_LEA signed integer.36signedLittle Endianuint37_BEAn unsigned integer.37unsignedBig Endianuint37_LEAn unsigned integer.37unsignedLittle Endianint37_BEA signed integer.37signedBig Endianint37_LEA signed integer.37signedLittle Endianuint38_BEAn unsigned integer.38unsignedBig Endianuint38_LEAn unsigned integer.38unsignedLittle Endianint38_BEA signed integer.38signedBig Endianint38_LEA signed integer.38signedLittle Endianuint39_BEAn unsigned integer.39unsignedBig Endianuint39_LEAn unsigned integer.39unsignedLittle Endianint39_BEA signed integer.39signedBig Endianint39_LEA signed integer.39signedLittle Endianuint40_BEAn unsigned integer.40unsignedBig Endianuint40_LEAn unsigned integer.40unsignedLittle Endianint40_BEA signed integer.40signedBig Endianint40_LEA signed integer.40signedLittle Endianuint41_BEAn unsigned integer.41unsignedBig Endianuint41_LEAn unsigned integer.41unsignedLittle Endianint41_BEA signed integer.41signedBig Endianint41_LEA signed integer.41signedLittle Endianuint42_BEAn unsigned integer.42unsignedBig Endianuint42_LEAn unsigned integer.42unsignedLittle Endianint42_BEA signed integer.42signedBig Endianint42_LEA signed integer.42signedLittle Endianuint43_BEAn unsigned integer.43unsignedBig Endianuint43_LEAn unsigned integer.43unsignedLittle Endianint43_BEA signed integer.43signedBig Endianint43_LEA signed integer.43signedLittle Endianuint44_BEAn unsigned integer.44unsignedBig Endianuint44_LEAn unsigned integer.44unsignedLittle Endianint44_BEA signed integer.44signedBig Endianint44_LEA signed integer.44signedLittle Endianuint45_BEAn unsigned integer.45unsignedBig Endianuint45_LEAn unsigned integer.45unsignedLittle Endianint45_BEA signed integer.45signedBig Endianint45_LEA signed integer.45signedLittle Endianuint46_BEAn unsigned integer.46unsignedBig Endianuint46_LEAn unsigned integer.46unsignedLittle Endianint46_BEA signed integer.46signedBig Endianint46_LEA signed integer.46signedLittle Endianuint47_BEAn unsigned integer.47unsignedBig Endianuint47_LEAn unsigned integer.47unsignedLittle Endianint47_BEA signed integer.47signedBig Endianint47_LEA signed integer.47signedLittle Endianuint48_BEAn unsigned integer.48unsignedBig Endianuint48_LEAn unsigned integer.48unsignedLittle Endianint48_BEA signed integer.48signedBig Endianint48_LEA signed integer.48signedLittle Endianuint49_BEAn unsigned integer.49unsignedBig Endianuint49_LEAn unsigned integer.49unsignedLittle Endianint49_BEA signed integer.49signedBig Endianint49_LEA signed integer.49signedLittle Endianuint50_BEAn unsigned integer.50unsignedBig Endianuint50_LEAn unsigned integer.50unsignedLittle Endianint50_BEA signed integer.50signedBig Endianint50_LEA signed integer.50signedLittle Endianuint51_BEAn unsigned integer.51unsignedBig Endianuint51_LEAn unsigned integer.51unsignedLittle Endianint51_BEA signed integer.51signedBig Endianint51_LEA signed integer.51signedLittle Endianuint52_BEAn unsigned integer.52unsignedBig Endianuint52_LEAn unsigned integer.52unsignedLittle Endianint52_BEA signed integer.52signedBig Endianint52_LEA signed integer.52signedLittle Endianuint53_BEAn unsigned integer.53unsignedBig Endianuint53_LEAn unsigned integer.53unsignedLittle Endianint53_BEA signed integer.53signedBig Endianint53_LEA signed integer.53signedLittle Endianuint54_BEAn unsigned integer.54unsignedBig Endianuint54_LEAn unsigned integer.54unsignedLittle Endianint54_BEA signed integer.54signedBig Endianint54_LEA signed integer.54signedLittle Endianuint55_BEAn unsigned integer.55unsignedBig Endianuint55_LEAn unsigned integer.55unsignedLittle Endianint55_BEA signed integer.55signedBig Endianint55_LEA signed integer.55signedLittle Endianuint56_BEAn unsigned integer.56unsignedBig Endianuint56_LEAn unsigned integer.56unsignedLittle Endianint56_BEA signed integer.56signedBig Endianint56_LEA signed integer.56signedLittle Endianuint57_BEAn unsigned integer.57unsignedBig Endianuint57_LEAn unsigned integer.57unsignedLittle Endianint57_BEA signed integer.57signedBig Endianint57_LEA signed integer.57signedLittle Endianuint58_BEAn unsigned integer.58unsignedBig Endianuint58_LEAn unsigned integer.58unsignedLittle Endianint58_BEA signed integer.58signedBig Endianint58_LEA signed integer.58signedLittle Endianuint59_BEAn unsigned integer.59unsignedBig Endianuint59_LEAn unsigned integer.59unsignedLittle Endianint59_BEA signed integer.59signedBig Endianint59_LEA signed integer.59signedLittle Endianuint60_BEAn unsigned integer.60unsignedBig Endianuint60_LEAn unsigned integer.60unsignedLittle Endianint60_BEA signed integer.60signedBig Endianint60_LEA signed integer.60signedLittle Endianuint61_BEAn unsigned integer.61unsignedBig Endianuint61_LEAn unsigned integer.61unsignedLittle Endianint61_BEA signed integer.61signedBig Endianint61_LEA signed integer.61signedLittle Endianuint62_BEAn unsigned integer.62unsignedBig Endianuint62_LEAn unsigned integer.62unsignedLittle Endianint62_BEA signed integer.62signedBig Endianint62_LEA signed integer.62signedLittle Endianuint63_BEAn unsigned integer.63unsignedBig Endianuint63_LEAn unsigned integer.63unsignedLittle Endianint63_BEA signed integer.63signedBig Endianint63_LEA signed integer.63signedLittle Endianuint64_BEAn unsigned integer.64unsignedBig Endianuint64_LEAn unsigned integer.64unsignedLittle Endianint64_BEA signed integer.64signedBig Endianint64_LEA signed integer.64signedLittle Endianfloat32_LEA signed floating point number.32signedLittle Endianfloat32_BEA signed floating point number.32signedBig Endianfloat64_LEA signed floating point number.64signedLittle Endianfloat64_BEA signed floating point number.64signedBig Endianboolean8_LEA boolean. "1" means TRUE; "0" FALSE.8N/ALittle EndianCaveatsThe namestringis special. Any type in the database with the type name ofstringis assumed to be a string
and parsed as such by the XTCE tool. This means donotdo things like this in code:struct string
{
  char buffer[64];
};unsignedis assumed to be the same asunsigned intas perc++ standards.YAMCS-XTCE QuirksOur xtce flavor adheres to yamcs. In the future, we'll try our best to design our toolchain in such a way we can
be 100% compliant withxtce, but at the moment we adhere to yamcs-flavored xtceonly. As a result of this, we will
need to deal with any quirks this yamcs-flavored xtce standard may have. Below are a list of quirks we've discovered
so far:It seems that yamcs has issues processing anyParamterentries that have a '#' in the name.Notes For DevelopersThis project can also be used as a library and devs can obtain it from Pypi(particularly useful to parse XTCE files):pip install xtce-generatorMakesureyou havelxml>=4.6installed. Otherwise some of the parse methods will not work properly onxtcemodule.TestingAll testing of this tool is done withauto-yamcsDocumentation updated on April 5, 2022