# Presentation: 316857

## pyExpressa Python Open Source EXPRESS Compiler / Code Generatorin Progress

- Hans Peter de Koning (ESA/ESTEC D/TOS-MVC, The Netherlands)
- Hans-Peter.de.Koning@esa.int
- 2003 NASA-ESA Workshop on Aerospace Product Data Exchange
- April 7-10, 2003, NIST, Gaithersburg, Md, USA

**Notes:**
<Simulog est une société fondée à l’initiative de l’INRIA (INSTITUT NATIONAL DE RECHERCHE EN INFORMATIQUE ET AUTOMATIQUE) il y a plus de 10 ans, en 1984.
La définition de la mission de l’entreprise  est basée sur l’objectif initial de ses initiateurs: être un pont entre l’industrie et la recherche publique.

La mission de l’entreprise comporte deux points qui reflètent notre stratégie mais aussi le marché sur lequel Simulog évolue:
L’objectif numéro 1 est de fournir une gamme de prestation complète: Simulog doit intégrer les solutions proposées dans l’environnement industriel préexistant de ses clients.
Cette connaissance approfondie des besoins du client , de son métier et de son environnement permet d’atteindre plusieurs objectifs positifs: 1. la mise en confiance du client, qui facilite le dialogue lors de la phase de spécification des besoins (meilleure compréhension des besoins) et lors de l’apport des solutions nouvelles (plus grande facilité à obtenir l’adhésion du client). 2. la fidélisation du client, qui permet d'accroître l’efficacité et la rentabilité des études au fur et à mesure.
L’objectif numéro 2 consiste à s’adapter aux nouvelles technologies pour offrir au client une solution industrielle à un problème ne pouvant être traité par les méthodes classiques.
>

## Why another EXPRESS compiler? (1)

- Some lessons learned from past STEP-TAS converter developments:
- Support of STEP-TAS library on multiple platform/compiler combinations (PC/Windows, Unixes) using COTS EXPRESS development kits:
- created distribution and maintenance trouble and is costly
- some combinations not supported by run-time libraries
- long feedback / improvement cycles between converter implementers and library software providers
- Late binding or SDAI implementations lead to slow, memory and CPU hungry applications and to unnecessarily complicated programming
- Provision of an application protocol together with a software library to facilitate implementation of the protocol is very good
- Manually crafted libraries are cumbersome and error-prone to create and maintain - investments in code generation wherever possible pay off hugely

- 2003 NASA-ESA Workshop on Aerospace PDE at NIST

- 7-10 April 2003

## Why another EXPRESS compiler? (2)

- Regarding implementation of STEP data exchange, ESA wants:
- to lower the threshold for implementation as much as possible
- to ensure the availability of data exchange solutions in the long term
- to avoid any dependency on third party run-time licenses
- to ensure full portability across all computer platforms / compiler combinations used for (space) analysis tools
- Examples from XML world show value of having good OSS tools
- Great asset in getting standards based data exchange disseminated
- There exist some open source EXPRESS tools
- but they are either immature
- or use programming languages not commonly used in industry

- 2003 NASA-ESA Workshop on Aerospace PDE at NIST

- 7-10 April 2003

## pyExpress development history

- June 2002: Decided to prototype an EXPRESS compiler and library code generator in Python for STEP-NRF and STEP-TAS
- September 2002: Completed feasibility study (~250h) successfully
- Basic scanner / parser / Python library generator working
- Based on the OSS pure Python SPARK lexer/parser package
- see http://pages.cpsc.ucalgary.ca/~aycock/spark
- October 2002: Decided to go ahead with pyExpress code generators for industrial use in upgraded STEP-TAS libraries and converters
- Simulog & Graitec (France): ANSI C++ library generator, C/C++ converters
- ESA/ESTEC: Python class library generator, TASverter tool
- March 2003: First releases of pyExpress, TASverter, C++ libraries

- 2003 NASA-ESA Workshop on Aerospace PDE at NIST

- 7-10 April 2003

## pyExpress objectives

- Provide a STEP converter development environment, that
- can be used by converter developer with minimal STEP knowledge
- can be used as a Rapid Application Development tool for prototyping and near-real time validation and refinement of application protocols
- has very strong string manipulation capabilities
- maps well onto EXPRESS object oriented data models
- leads to industrially robust converters, with acceptable performance and memory requirements, also for large models
- In first release put focus on immediate needs:
- Support long form schemas, CONSTANTs, TYPEs, ENTITYs, FUNCTIONs
- For ENTITYs only explicit, redeclared and DERIVEd attributes and WHERE rules, but not yet INVERSE attributes and UNIQUE constraints

- 2003 NASA-ESA Workshop on Aerospace PDE at NIST

- 7-10 April 2003

## pyExpress design approach

- Highly modular
- Support ISO 10303-11:1994 (i.e. EXPRESS version 1)
- Support ISO 10303-21:1994 (single schema / data section)
- Generate early binding class libraries
- Start with Python and C++
- Stay as close as possible to EXPRESS schema structure, but adapt to target language conventions
- Make use of powerful Python (class / type) reflection capabilities
- Defensive programming and runtime type checking throughout
- Python is a weakly typed language
- Tool is too large to rely on programmer discipline: use positive asserts

- 2003 NASA-ESA Workshop on Aerospace PDE at NIST

- 7-10 April 2003

## pyExpress architecture

- Scanner /
- Tokenizer

- Parser

- Semantic
- Analyzer

- Python
- Code
- Generator

- C++
- Code
- Generator

- EXPRESS
- schema A

- SPARK

- EXPRESS
- MetaModel
- Handler

- Early binding
- Python
- class library
- for schema A

- Early binding
- C++
- class library
- for schema A

- tokenlist

- abstract syntax tree

- uses

- pyExpress
- utils

- pyExpress

- HTML
- API manual
- for Python
- class library A

- 2003 NASA-ESA Workshop on Aerospace PDE at NIST

- 7-10 April 2003

## Outline of converter based on pyExpress

- Outline of converter based on pyExpress

- Tool X to
- Schema A
- reader

- Generic
- STEP repository
- handler

- File or
- in-memory data
- for Tool X

- Converter X-A-Y

- Support library
- for Schema A:
- macro &
-  convenience
- functions

- Schema A
- to Tool Y
- writer

- ISO 10303-21
- file
- for dataset
- conforming to
- Schema A

- Part 21
- reader

- Part 21
- writer

- Generated
- class library
- for Schema A

- File or
- in-memory data
- for Tool Y

- output or
- input

- contains
- in-memory dataset(s)
- conforming to Schema A

- output

- input

- Python or C++ API for Schema A

- ‘Schema A Library provided by pyExpress’

- To be produced
- by converter
- developer

- 2003 NASA-ESA Workshop on Aerospace PDE at NIST

- 7-10 April 2003

## Design principlesEXPRESS to Python mapping (1)

- Generate human readable Python
- Close resemblance between EXPRESS ENTITY and Python class
- attributes one-to-one mapping, Java Bean type get, set accessors
- use same class hierarchy (SUBTYPE OF  Python subclassing)
- All Python classes inherit from ExpressEntity root class
- ExpressEntity takes care of all generic attributes and methods, like writing itself to Part 21 format or reconstructing itself from Part 21 format
- Built-in automatic storage and registration into the ‘active’ ExpressDataSet upon class constructor calls
- An intuitive ‘pythonic’ reference API: all pointers are normal Python references, no need to explicitly manipulate instance IDs or pointers
- Dynamic extension / filling of aggregates: SET, BAG, LIST, ARRAY

- 2003 NASA-ESA Workshop on Aerospace PDE at NIST

- 7-10 April 2003

## Design principlesEXPRESS to Python mapping (2)

- Generate explicit (run-time called) type checking functions
- For all types defined in EXPRESS SCHEMA, including aggregation types
- Called in assert statements as pre-conditions in all relevant functions/methods
- DERIVE / INVERSE attributes, WHERE rules  service methods
- on all generated Python classes
- check_where_rules() executed on demand, for selected rule(s) or all
- FUNCTIONSs, PROCEDUREs, RULEs  Python def’s
- Algorithmic code converted statement by statement to Python equivalent
- Note: RULEs not implemented yet
- EXPRESS built-in functions  manually crafted Python equivalents
- Elegant mapping for SELECT TYPEs  Python polymorphic ref’s

- 2003 NASA-ESA Workshop on Aerospace PDE at NIST

- 7-10 April 2003

## EXPRESS  Python Example (1)

- 2003 NASA-ESA Workshop on Aerospace PDE at NIST

- 7-10 April 2003

## EXPRESS  Python Example (2)

- 2003 NASA-ESA Workshop on Aerospace PDE at NIST

- 7-10 April 2003

## EXPRESS  Python Example (3)HTML API documentation

- Any parameter
- can be None
- upon creation
- of an instance.
- Values can be
- given later using
- the class’
- set_ methods.
- Creating an instance
- automatically
- registers it in the
- ‘active’ dataset.

- 2003 NASA-ESA Workshop on Aerospace PDE at NIST

- 7-10 April 2003

## Main benefits from pyExpress generated Python binding

- Easy-to-use, clean Rapid Converter Development tool
- Very valuable for protocol validation
- Regenerating a complete library takes only a few minutes after changing the EXPRESS schema
- Python has an extremely friendly build / error trapping environment
- No need for makefiles, explicit compile or link steps: just modify the source and rerun a test case
- Turn-around time on bugs is usually in the order of a few minutes
- Full availability of Python’s very powerful string manipulation and modules
- Good interfacing with C/C++ modules (if needed for speed or i/f existing code)
- Fast enough for real-world applications
- Datasets of 10000 instances convert in <1 minute on typical PC hardware

- 2003 NASA-ESA Workshop on Aerospace PDE at NIST

- 7-10 April 2003

## Current limitations of pyExpressPython code generator

- Not all EXPRESS expressions converted correctly
- Some EXPRESS built-ins don’t work yet: e.g. QUERY
- WHERE rule checking not fully working
- INVERSE and RULE not implemented yet
- GENERIC type not handled yet
- Only works on long form schemas: USE, REF FROM not supported
- Complex instances not implemented yet
- Part 21 reader only prototyped, not fully functioning

- 2003 NASA-ESA Workshop on Aerospace PDE at NIST

- 7-10 April 2003

## pyExpress C++ code generator

- Developed by Simulog & GRAITEC (France) as additional pyExpress back-end: code generator targets pure ANSI C++
- Incorporates GRAITEC’s fast Part 21 reader/writer: 50,000 instances / minute on typical PC hardware
- First full delivery received March 2003
- Verified to work correctly as replacement in existing converters
- ESARAD/STEP-TAS and TRASYS/STEP-TAS implemented in Coriginally on top of an ST-Developer SDAI library

- 2003 NASA-ESA Workshop on Aerospace PDE at NIST

- 7-10 April 2003

## Main benefits from pyExpress generated C++ binding

- Clean portable, early-binding C++ implementation
- Closely follows EXPRESS schema, ENTITY hierarchy and TYPEs
- Good interfacing with existing C++ or C codes
- Very good performance on Part 21 files

- 2003 NASA-ESA Workshop on Aerospace PDE at NIST

- 7-10 April 2003

## Future of pyExpress

- Will be published as open source by ESA
- ESA contract with Manchester Informatics Ltd. (UK) starts in a few weeks to upgrade the compiler’s capabilities and formal correctness
- University of Manchester (UK) / Group of Prof. Hilary Kahn
- Based on their JEX tool (EXPRESS model processing using Java/Python)
- Implement and verify full support for EXPRESS v1
- In use by GRAITEC (and building/construction partners) in frame of developing IFC converters
- Upgrade to process directly an HTML or XML version of a STEP application protocol (ARM or AIM clause), automatically generating a fully documented library

- 2003 NASA-ESA Workshop on Aerospace PDE at NIST

- 7-10 April 2003

## Acknowledgements

- Co-workers at ESA/ESTEC: Simon Appel, David Alsina
- Contractor Simulog (France): Olivier Pailles, Arnaud Klinger
- Subcontractor GRAITEC (France): Eric Lebègue
- The developers of exp-engine.sourceforge.net (expresso):	Peter Denno, Craig Lanning, Steve Waterburyand osexpress.sourceforge.net:	Josh Lubell, Michael Keenan, Stephane Lardetfor valuable ideas and examples on how to parse EXPRESS schemas
- The developer of the Python SPARK:	John Aycock (http://pages.cpsc.ucalgary.ca/~aycock/spark)

- 2003 NASA-ESA Workshop on Aerospace PDE at NIST

- 7-10 April 2003