big commit, changing the scanning functions to use a structure instead of global variables


git-svn-id: https://foundry.supelec.fr/svn/luatex/branches/tokenlib@4977 0b2b3880-5936-4365-a048-eb17d2e5a6bf
remove the no_new_control_sequence variable (use a function argument instead)


git-svn-id: https://foundry.supelec.fr/svn/luatex/branches/tokenlib@4978 0b2b3880-5936-4365-a048-eb17d2e5a6bf
trying to remove scanner_status (work in progress)


git-svn-id: https://foundry.supelec.fr/svn/luatex/branches/tokenlib@4979 0b2b3880-5936-4365-a048-eb17d2e5a6bf
get scanner_status out of align.w


git-svn-id: https://foundry.supelec.fr/svn/luatex/branches/tokenlib@4980 0b2b3880-5936-4365-a048-eb17d2e5a6bf
still trying to get rid of scanner_status (probably won't work, though :()



git-svn-id: https://foundry.supelec.fr/svn/luatex/branches/tokenlib@4981 0b2b3880-5936-4365-a048-eb17d2e5a6bf
undo 4979--4981

git-svn-id: https://foundry.supelec.fr/svn/luatex/branches/tokenlib@4982 0b2b3880-5936-4365-a048-eb17d2e5a6bf
prevent some files from using |buffer|, and cleanup id lookups


git-svn-id: https://foundry.supelec.fr/svn/luatex/branches/tokenlib@4983 0b2b3880-5936-4365-a048-eb17d2e5a6bf
playing with the expanded code code a bit


git-svn-id: https://foundry.supelec.fr/svn/luatex/branches/tokenlib@4984 0b2b3880-5936-4365-a048-eb17d2e5a6bf
start an attempt to clean up the input stack processing


git-svn-id: https://foundry.supelec.fr/svn/luatex/branches/tokenlib@4985 0b2b3880-5936-4365-a048-eb17d2e5a6bf
change some bare numbers into #define's, and rename a few pseudo input functions


git-svn-id: https://foundry.supelec.fr/svn/luatex/branches/tokenlib@4986 0b2b3880-5936-4365-a048-eb17d2e5a6bf
tiny reformat


git-svn-id: https://foundry.supelec.fr/svn/luatex/branches/tokenlib@4987 0b2b3880-5936-4365-a048-eb17d2e5a6bf
convert \scantokens to use strings instead of nodes


git-svn-id: https://foundry.supelec.fr/svn/luatex/branches/tokenlib@4992 0b2b3880-5936-4365-a048-eb17d2e5a6bf
small c99 fix


git-svn-id: https://foundry.supelec.fr/svn/luatex/branches/tokenlib@4993 0b2b3880-5936-4365-a048-eb17d2e5a6bf
consolidate various file info objects into a single array of structures


git-svn-id: https://foundry.supelec.fr/svn/luatex/branches/tokenlib@4994 0b2b3880-5936-4365-a048-eb17d2e5a6bf
fix a subtle misuse of line_catcode_table


git-svn-id: https://foundry.supelec.fr/svn/luatex/branches/tokenlib@4995 0b2b3880-5936-4365-a048-eb17d2e5a6bf
scan_toks can save scanner_status and warning_index itself


git-svn-id: https://foundry.supelec.fr/svn/luatex/branches/tokenlib@4996 0b2b3880-5936-4365-a048-eb17d2e5a6bf
the return value of scan_toks() was almost never used, make that really never. 
this opens up the way to have scan_toks() return def_ref, instead


git-svn-id: https://foundry.supelec.fr/svn/luatex/branches/tokenlib@4997 0b2b3880-5936-4365-a048-eb17d2e5a6bf