- Moved some ad-hoc tests from the Program module of fsharplex into the FSharpLex.Tests project.
- Modified NFA and DFA generation code so they'll work for lexer rules instead of just Regexes.
- In-progress commit while reworking our code to use regular expression derivatives; this will ultimately provide much better support for character sets (especially Unicode).
- Implemented most of the regex canonicalization code.
- Moved the cases from the CharacterClass type into Regex/ExtendedRegex.
- Implemented some code which determines which characters belong to which Unicode categories; we'll use this later to efficiently implement the Regex.Any case and Unicode category patterns in the lexer rules.
- Implemented some additional functions in the CharSet module which are used when canonicalizing a Regex.
- Added documentation comments to the functions in the CharSet module.
- Removed the tokenizer code; it'll be moved into a library project later (for anyone that wants to use a table-driven lexer).
- Removed the dead/unused 'reverse' and 'minimize' functions -- these aren't needed anymore since the new derivative-based algorithm produces a close-to-minimal DFA.
- Implemented more Regex simplification/canonicalization rules.
- Added a small optimization to the CharSet.addRange function for when the input set is empty.
- Implemented algorithm for computing approximate derivative classes, but it's not quite right yet -- the use of Set.intersection seems to be messing things up and causing the result to always be empty.
- Implemented some additional 'fold' functions in the CharSet module which fold over the intervals themselves instead of the elements in the intervals; this should provide a performance benefit in some usage scenarios.
- Fixed the computation of the derivative classes for a compound expression so it matches the equation from the paper.
- Created a RegularVector module and implemented some functions for working with regular vectors.
- Deleted the Nfa module and the LexerNfaGraph type from the Graph module because the new derivative-based algorithm constructs a DFA directly from the Regex.
- Commented out most of the code in the Dfa module until it can be updated to work with the new algorithm.
- Committing progress made on getting the DFA compilation back to a working state.
- NOTE: This commit contains one error which causes the code not to compile.
- Reworked the DFA-compilation algorithm to use a worklist-based algorithm like the earlier NFA-to-DFA compilation algorithm.
- Moved the ExtendedRegex type into the Ast module and renamed it to LexerPattern.
- Added new cases to LexerPattern to handle user-defined pattern macros and Unicode categories.
- Moved the code which creates the UnicodeCategory to CharSet map into the Ast module and placed it in a new Unicode submodule.
- Initialize the dictionary with the necessary capacity so it won't need to be resized.
- Created placeholder modules for the lexer and parser which we'll eventually add.
- Moved the function used within the 'fold' in 'createDfa' out to a separate function to make the code easier to read.
- Implemented the RegularVector.canonicalize function and used to to ensure any regular vectors are canonicalized before they're used.