Talk:Lexical analysis

message: ----------------------
This article is not clear enough! It needs more examples!
Robert A.
----------------------
The link to ocaml-ulex  in 'Links' is broken.
Frank S.
Robert, this article is actually very poorly articulated. It seems that its authors do not have enough clarity of how to explain it, for it to be comprehensible. And the article should probably be marked that way - that is requires further clarity.Stevenmitchell (talk) 16:08, 25 January 2014 (UTC)
thread_title: Examples

message: Can someone explain to me what types of tokens there are? (Keywords, identifiers, literals? Or are there more) And do each of these types go into a symbol table of that type? i.e. an identifier table, keyword table and literal table? Or are they just stored into one uniform symbol table? —Dudboi 02:17, 6 November 2006 (UTC)
thread_title:  Types of tokens 

message: I've ``merged`` token (parser) here. The page is a bit of a mess now though. The headings I've made should help sort that out. The examples should be improved so that they take up less space. The lex file should probably be moved to the flex page. --MarSch 15:37, 30 April 2007 (UTC)
replies: text: done the move of the example. --MarSch 15:39, 30 April 2007 (UTC)
thread_title:  merger and clean up 

message: It would be nice to see what is involved in the next step, or at least see a link to the page describing the whole process of turning high level code into low level code
-Dusan B.
:you mean compiling? --MarSch 10:53, 5 May 2007 (UTC)
thread_title:  Next? 

message: There's a mention that scanning fails on an "invalid token". It doesn't seem particularly clear what constitutes a lexical error other than a string of garbage characters. Any ideas? --138.16.23.227 (talk) 04:54, 27 November 2007 (UTC)
replies: text: The comment needs some context.  Generally speaking, a lexical analyzer may report an error.  But that is usually (for instance in Lex programming tool) under control of the person who designs the rules for the analysis.  The analyzer itelf may reject the rules because they're inconsistent.  On the other hand, the parser is more likely to have built-in behavior -- yacc for instance fails on any mismatch, requires the developer to specify how to handle errors (updating the article to reflect these comments requires citing reliable sources of course - talk pages aren't a source) Tedickey (talk) 13:56, 27 November 2007 (UTC)
thread_title:  Lexical Errors 

message: There should be some more clearly explaned what exactly differs those modules, what are their particular jobs and how they're composed together (maybe some illustration?)  —Preceding unsigned comment added by Sasq777 (talk • contribs) 16:06, 11 June 2008 (UTC)
thread_title:  Lexer, Tokenizer, Scanner, Parser 

message: In the article, there are two examples I want to comment on. One shows XML. Another uses an s-expression. These examples create the impression that lexing produces a tree data structure. However, lexical analysis generates a list of tokens (often tuples). To be clear, a list, not a tree. In the lexers I've used, tokens do not contain other tokens. There may be exceptions, but if so, they are uncommon.
To show lexing, it is better to use a (simple) list data structure. Below is an example in Python. (JSON would also be suitable. The idea is to choose a commonly used data representation to get the point across; namely, you only need a list of tuples.)
from enum import Enum
class Token(Enum):
    WORD = 1
    PUNC = 2
tokenization = [
  (Token.WORD, "The"),
  (Token.WORD, "quick"),
  (Token.WORD, "brown"),
  (Token.WORD, "fox"),
  (Token.WORD, "jumps"),
  (Token.WORD, "over"),
  (Token.WORD, "the"),
  (Token.WORD, "lazy"),
  (Token.WORD, "dog"),
  (Token.PUNC, "."),
]
DavidCJames (talk) 15:45, 10 March 2021 (UTC)
thread_title:  The lexing example should use a list, not XML nor s-expressions 

message: It seems to be a duplicate of List of parser generators.  Probably can be removed? 98.176.182.199 (talk) 02:57, 6 December 2009 (UTC)
replies: text: It's not a duplicate (as the casual reader will observe). Tedickey (talk) 12:00, 6 December 2009 (UTC)
thread_title:  Regarding "Lexical analyzer generators" section 

message: I am not familiar with the code examples in SICP, can someone please
provide the section/example name/number and possibly a link to the example.
Thanks,
Ckoch786 (talk) 01:43, 7 February 2014 (UTC)
thread_title:  SICP book example 

message: A recent edit, using the sense that recursion is good, and popular means the same thing equated popular with support for recursive regular expressions.  We need a reliable source of information discussing that aspect and which design features directly support recursion TEDickey (talk) 11:17, 1 January 2017 (UTC)
thread_title:  recursion vs regular expressions 

message: The software section in this article was removed some time ago by consensus, which was a good decision because there is already a link to separate Wikipedia page with a list of implementations and sufficient details. It seems it is put back (?) with some rather specific examples instead of trying to be generic. Worse (in my opinion), there are claims in that section such as "Lex is too complex and inefficient" that clearly do not belong there. I will remove this section unless someone has a strong argument why it should be included.  — Preceding unsigned comment added by Robert van Engelen (talk • contribs) 13:24, 29 April 2021 (UTC)
thread_title:  Section on Software 

message: Is there a Wikipedia article that discusses the kind of tokenization used many BASIC dialects?
My understanding is:
* (a) When typing in lines of a program or loading a program from disk/cassette, common keywords such as "PRINT", "IF(" (including the opening parenthesis), "GOSUB", etc. are recognized and stored as a single byte or two in RAM, some or all spaces were discarded, line numbers and internal decimal numbers are converted to internal binary format(s), etc.
* (b) When listing the program with LIST or storing a program to disk/cassette, those tokens are expanded back to the full human-readable name of the keyword, spaces were inserted as necessary (pretty-printing), those binary values are printed out in decimal, etc.
This tokenization (lexical analysis) article
covers the sort of thing done in compilers similar to (a),
but that article says nothing about (b), which compilers never do,
but practically every BASIC dialect mentioned in the "List of computers with on-board BASIC" article does.
The source-code editor article briefly mentions "tokenizing editors" that do both (a) and (b).
I've heard that many Forth language implementations have a "see" command that does something similar to (b).
The BASIC interpreter#Tokenizing and encoding lines article and most of our articles on specific BASIC implementations (GFA BASIC, GW-BASIC, Atari BASIC#Tokenizer, etc.) briefly discuss such tokenization.
But I still wonder -- do any other languages typically have implementations that do such tokenization?
Should *this* "tokenization (lexical analysis)" article have a section at least mentioning (b), or is there already some other article that discusses it?
(I'm reposting this question that I previously posted at Talk:BASIC#tokenization, because it seems more widely applicable than just the BASIC language.
).
--DavidCary (talk) 19:19, 27 September 2023 (UTC)
thread_title:  tokenizing editors 

message: — Assignment last updated by Minhngo6 (talk) 18:32, 7 March 2024 (UTC)
thread_title: Wiki Education assignment: Linguistics in the Digital Age