Patent Publication Number: US-6993752-B2

Title: Compiler with dynamic lexical scanner adapted to accommodate different character sets

Description:
FIELD OF THE INVENTION 
   The invention relates generally to compiling of source code, and more specifically, to a compiler with a lexical scanner adapted to handle different character sets. 
   BACKGROUND OF THE INVENTION 
   The application has particular, though not exclusive, application to compiling of C source code. A C compiler must normally handle the Standard C Character Set which consists of at least 91 characters (upper and lower case alphabetic characters, numbers, and symbols representing various operators). The characters are encoded in the source code with numeric values or “code points.” Code points representing the complete sets of valid C characters are generally referred to as “character coded set identifiers” (CCSIDs). During initial phases of compilation, a lexical scanner module usually converts raw source code into corresponding tokenized target code. The lexical scanner relies on a character identification table embodying a particular CCSID to interpret the source code and identify individual characters. 
   The lexical scanner associated with a C compiler expects to receive source code associated with a particular character set. However, various CCSIDs are used to encode the Standard C Character Set, and source code incorporating one character set must be converted before the code may be compiled with a compiler that uses a different character set. Utilities are commonly available to convert C source code from one standard character set to another. 
   It would be desirable to provide a lexical scanner that can adapt itself dynamically to scan source code in different standard character sets and to produce corresponding target code in different standard character sets. 
   SUMMARY OF THE INVENTION 
   In one aspect, in a compiler with a lexical scanner that uses a character identification table to convert source code into corresponding target code, the invention provides a method of dynamically accommodating different character sets associated with source and target codes. Data is stored to allow conversion of either the source code or the table from one standard character set to another. Before actual scanning begins, the character set associated with the source code is compared with the target character set. If the character sets are different, the source code is translated to conform to the target character set. Similarly, the character set associated with the character identification is compared with the target character set. If the character sets are different, the table is converted to conform to the target character set. The dynamic nature of the character identification table allows the user to specify target code that uses a desired character set. 
   The term “character set” as used in this specification should be understood as encompassing a notional set of human intelligible characters in combination with numeric values (code points) that uniquely encode or identify the character. An item such as source code, target code, or a character identification table is “associated with,” “converted to,” or “translated to” a specific character set if the contents of the item are encoded using the point values of the character set. 
   Other aspects of the invention will be apparent from a description of a preferred embodiment below and will be more specifically defined in the appended claims. 

   
     DESCRIPTION OF THE DRAWINGS 
     The invention will be better understood with reference to drawings in which: 
       FIG. 1  is a diagrammatic representations of a dynamic lexical scanner; 
       FIG. 2  is a flowchart illustrating a method of accommodating different character sets during lexical scanning. 
   

   In the flowcharts, the direction of program flow is down the page and away from decision boxes unless otherwise indicated with arrows. 
   DESCRIPTION OF THE PREFERRED EMBODIMENTS 
   Reference is made to  FIG. 1  which diagrammatically illustrates a lexical scanning module  10  associated with a C compiler. The scanning module  10  receives C source code  12  from a disk file and produces tokenized C target code  14 , which may be stored as a disk file. For purposes of discussion, it will be assumed that the source code  12  is associated with a character set identified as EBCDIC 00037 and the target code  14 , with a character set identified as EBCDIC 00500, both IBM™ standards. Minor discrepancies exist between the two character sets. For example, the open bracket associated with a C-type indexing operator “[” is represented by the code point XXX in EBCDIC 00037 and the code point YYY in EBCDIC 00500. The present compiler is adapted in a conventional manner to permit a user to specify, as part of a desired target, the target character set to be used. 
   The lexical scanning module  10  relies on a character identification table  16  to interpret characters encoded in the source code  12 . The table  16  is simply an array relating code points to specific characters of the Standard C Character Set. Unlike the prior art, the table  16  is dynamic, its contents being adjusted automatically, as discussed more fully below, to accommodate different target character sets. The information, specifically entry values, required to convert the table  16  to a desired character set is stored among conversion data  18 , on disk. In its simplest form, the stored information may consist of distinct arrays, each associated with a different EBCDIC standard that the compiler is required to handle. The contents of the character identification table  16 , which may be stored in RAM during operation, may simply be replaced with an appropriate array. 
   The conversion data  18  may also comprise several arrays for use in translating the source code  12  to any one of several different EBCDIC formats. Each entry of such an array effectively maps a code point of one EBCDIC format into another. With reference to the two EBCDIC formats specified above, the conversion array would, for example, map the value XXX representing the character “[” in EBCDIC 00037 into the value YYY used by EBCDIC 00500 to identify the same character. The entire source code  12  may be translated by loading the source code  12  into a buffer and replacing each code value in the source code  12  with the corresponding value extracted from the array. 
   How the scanning module  10  handles discrepancies in source and target character sets or user changes to specified target character sets will be described with reference to the flowchart of  FIG. 2 . During initialization, the compiler may load default values into the character identification table  16  (step  20 ) that correspond to a commonly used, default EBCDIC format. The character sets used by the source and targets codes  12 ,  14  are identified (step  22 ). The source character set (specifically the EBCDIC format associated with the source code  12 ) will normally be specified by the source file. The target character set may normally be set by the compiler to the default EBCDIC format, but, as mentioned above, the user may specify a different format when selecting a desired target. In any event, the character sets associated with the source code  12 , target code  14 , and the character identification table  16  are known or otherwise identified. 
   The scanning module  10  loads the source code  12  in a conventional manner from disk into a RAM buffer (step  24 ). The source character set is compared with the target character set (step  26 ). If the two character sets do not match, the buffered source code  12  is translated to the target character set: an appropriate translation array or table is retrieved from among the stored conversion data  18  (step  28 ), and each character value in the source buffer is replaced with a corresponding value extracted from the translation table (step  30 ). In this instance, a translation array mapping EBCDIC 00037 (source character set) into EBCDIC 00500 (target character sets) is used. The translation is conventional and standard routines may be used to perform the conversion. 
   The character set associated with the character identification table  16  is then compared with the target character set (step  32 ). If the two character sets do not match, the table  16  is converted to the target character set (step  32 ). Assuming that the user has changed target values, the character identification table  16  would be replaced with an array stored among the conversion data  18  that identifies characters associated with the target format, EBCDIC 00500. 
   Once the scanning module  10  has ensured that the buffered source code  12  and the character identification conform to the target character set, it processes the contents of the buffer on a character-by-character basis (steps  36 ,  38 ), producing the tokenized target code  14 . Such conversion of the source code  12  is entirely conventional and will not be discussed further. 
   The program code associated with the compiler may be provided to users together with conversion data  18  for handling various character sets or translating source files on processor-readable medium. Third parties may also provide conversion data on such media to allow users to expand the number of different character set formats that the compiler can accommodate. 
   It will be appreciated that particular embodiments of the invention have been described and that modifications may be made therein without departing from the spirit of the invention or necessarily departing from the scope of the appended claims.