Down with Emacs Lisp : Dynamic Scope Analysis

It is possible to translate code written in Emacs Lisp or another Lisp dialect which uses dynamic scoping to a more modern programming language with lexical scoping while largely preserving structure and readability of the code. The biggest obstacle to such an idiomatic translation from Emacs Lisp is the translation of dynamic binding into suitable instances of lexical binding: Many binding constructs in real programs in fact exhibit identical behavior under both dynamic and lexical binding. An idiomatic translation needs to detect as many of these binding constructs as possible and convert them into lexical binding constructs in the target language to achieve readability and efficiency of the target code. The basic prerequisite for such an idiomatic translation is thus a dynamic scope analysis which associates variable occurrences with binding constructs. We present such an analysis. It is an application of the Nielson/Nielson framework for flow analysis to a semantics for dynamic binding akin to Moreau’s. Its implementation handles a substantial portion of Emacs Lisp, has been applied to realistic Emacs Lisp code, and is highly accurate and reasonably efficient in practice. 1. MIGRATING EMACS LISP Emacs Lisp [16, 29] is a popular programming language for a considerable number of desktop applications which run within the Emacs editor or one of its variants. The actively maintained code base measures at around 1,000,000 loc. As the Emacs Lisp code base is growing, the language is showing its age: It lacks important concepts from modern functional programming practice as well as provisions for large-scale modularity. Its implementations are slow compared to mainstream implementations of other Lisp dialects. Moreover, the development of both Emacs dialects places The XEmacs package collection which includes many popular add-ons and applications currently contains more than 700,000 loc. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ICFP’01, September 3-5, 2001, Florence, Italy. Copyright 2001 ACM 1-58113-415-0/01/0009 . $5 00. comparatively little focus on significant improvements of the Emacs Lisp interpreter. On the other hand, recent years have seen the advent of a large number of extension language implementations of full programming languages suitable for the inclusion in application software. Specifically, several current Scheme implementations are technologically much better suited as an extension language for Emacs than Emacs Lisp itself. In fact, the official long-range plan for GNU Emacs is to replace the Emacs Lisp substrate with Guile, also a Scheme implementation [28]. The work presented here is part of a different, independent effort to do the same for XEmacs, a variant of GNU Emacs which also uses Emacs Lisp as its extension language. Replacing such a central part of an application like XEmacs presents difficult pragmatic problems: It is not feasible to reimplement the entire Emacs Lisp code base by hand. Thus, a successful migration requires at least the following ingredients: • Emacs Lisp code must continue to run unchanged for a transitory period. • An automatic tool translates Emacs Lisp code into the language of the new substrate, and it must produce maintainable code. Whereas the first of these ingredients is not particularly hard to implement (either by keeping the old Emacs Lisp implementation around or by re-implementing an Emacs Lisp engine in the new substrate), the second is more difficult. Even though a direct one-to-one translation of Emacs Lisp into a modern latently-typed functional language is straightforward by using dynamic assignment or dynamic-environment passing to implement dynamic scoping, it does not result in maintainable output code: Users of modern functional languages use dynamic binding only in very limited contexts such as exception handling or parameterization. As it turns out, the situation is not much different for Emacs Lisp users: For many lets and other binding constructs in real Emacs Lisp code, dynamic scope and lexical scope are identical ! Consequently, a good “idiomatic” translation of Emacs Lisp into, say, Scheme, should convert these binding constructs into the corresponding lexical binding constructs of the target substrate. The only problem is to recognize these binding constructs, or rather, distinguish those where the programmer “meant” dynamic scope from those where she “meant” lexical scope. Since with dynamic scope, bindings travel through the program execution much as values do, this requires a proper (let* ((filename (expand-file-name filename)) (file (file-name-nondirectory filename)) (dir (file-name-directory filename)) (comp (file-name-all-completions file dir)) newest) (while comp (setq file (concat dir (car comp)) comp (cdr comp)) (if (and (backup-file-name-p file) (or (null newest) (file-newer-than-file-p file newest))) (setq newest file))) newest)) Figure 1: Typical usage of let in Emacs Lisp. flow analysis. This paper presents such an analysis called dynamic scope analysis. Specifically, our contributions are the following: • We have formulated a semantics for a subset of Emacs Lisp, called Mini Emacs Lisp, similar to the sequential evaluation function for Λd by Moreau [20]. • We have applied the flow analysis framework of Nielson and Nielson [22] to the semantics, resulting in an acceptability relation for flow analyses of Mini Emacs Lisp programs. • We have used the acceptability relation to formulate and implement a flow analysis for Emacs Lisp which tracks the flow of bindings in addition to the flow of values. • We have applied the analysis to real Emacs Lisp code. More specifically, the analysis is able to handle mediumsized real-world examples with high accuracy and reasonable efficiency. The work presented here is a part of the el2scm project that works on the migration from Emacs Lisp to Scheme. However, the other aspects of the translation (such as front-end issues, correct handling of symbols, the code-data duality, treatment of primitives and so on) are outside the (lexical) scope of this paper. Indeed, the analysis could be used for a number of other purposes, among them the development of an efficient compiler for Emacs Lisp, or the translation to a different substrate such as Common Lisp. Overview.The next section presents some code examples which show the need for a dynamic scope analysis. Section 3 defines the syntax of Mini Emacs Lisp. Section 4 develops an operational semantics with evaluation contexts. Based on the semantics, Section 5 presents a specification of a correct flow analysis. The next section sketches a correctness proof. Our implementation approach is described in Section 7. Section 8 describes some experimental results gained with our implementation prototype. We end with a discussion of related work and a conclusion.