From: winter@vision.rs.ch (Martin Winter)
Subject: BUG: Umlaute in SEARCH
Reply-To: winter@vision.rs.ch
Organization: RadioSuisse Ltd, Berne, Switzerland
Date: Mon, 5 Oct 1992 10:31:46 GMT

(This problem exists at least since 6.0-3 until 6.1B4)

There is a bug (or feature ?) in the C runtime library. The toupper() and
tolower() only works with the standard 7-bit ASCII, not with the DEC
multinational character set.
But unfortunaly, the CLI-routines converts all entered command line 
parameters into uppercase, including the correct handling for the 8-bit
characters.

If you do a search, eg:
NEWS> search ha"lt             (a" as the umlaut)
then the CLI converts it to   SEARCH HA"LT   (still correct, all uppercase)
After that, the NEWS converts the search-argument back to lowercase to do 
the search, but uses the tolower() and gets:
   HA"LT  ->  hA"lt      (note: the umlaut is not translated back)
unfortunaly, this search is hopeless...

to fix this problem, I use the same way of translation as the CLI.
(I haven't found a system-call for that... is something availble for that?)
there are 2 routines to replace:

- strip_compress_lower()   (in NEWSRTL.C)
  old version:
--------------------------------------------------------------
void
strip_compress_lower(s)
  char *s;
{
  char *start = s,
       *r = s;

  while (*s) {
    if (isgraph(*s))
#if vax11c
      *r++ =tolower(*s);
#else
      { if (*s >= 'A' && *s <= 'Z') *r++ = *s - 'A' + 'a'; else *r++ = *s; }
#endif
    else if ((r > start) && (*(r-1) != ' ')) *r++ = ' ';
    s++;
    }
  if ((r > start) && (*(r-1) == ' ')) --r;
  *r = '\0';
}
---------------------------------------------------------------

  new version:  (replace the part above with this)
---------------------------------------------------------------
void
strip_compress_lower(s)
  char *s;
{
  char *start = s,
       *r = s;

  while (*s) {
    if (isgraph(*s))
/* 5-SEP-1992, Martin Winter   (winter@vision.rs.ch)
   The tolower does not convert Umlaute, but the CLI-routines does it.
   Because of that all Umlaute get wrong. We change that to the same
   way as the CLI-routines do.
*/
      if (((*s >= 'A') && (*s <= 'Z')) ||
          ((*s >= '\xC0') && (*s <= '\xCF')) ||
          ((*s >= '\xD1') && (*s <= '\xDD')))
        *r++ = *s - 'A' + 'a';
      else
        *r++ = *s;

    else if ((r > start) && (*(r-1) != ' ')) *r++ = ' ';
    s++;
    }
  if ((r > start) && (*(r-1) == ' ')) --r;
  *r = '\0';
}
---------------------------------------------------------------

- lower_case()    (in NEWSRTL.C)
  old version:
---------------------------------------------------------------
char *lower_case(s)
  char *s;
{
  register char *p = s;

  while (*p) {
#if vax11c
    *p = tolower(*p);
#else
    if (*p >= 'A' && *p <= 'Z') *p += 'a' - 'A' ;
#endif
    p++;
    }
  return(s);
}
---------------------------------------------------------------

  new version:  (replace the part above with this)
---------------------------------------------------------------
char *lower_case(s)
  char *s;
{
  register char *p = s;

  while (*p) {

/* 5-SEP-1992, Martin Winter   (winter@vision.rs.ch)
   The tolower does not convert Umlaute, but the CLI-routines do it.
   Because of that all Umlaute get wrong. We change that to the same
   way as the CLI-routines do.
*/
   if (((*p >= 'A') && (*p <= 'Z')) ||
       ((*p >= '\xC0') && (*p <= '\xCF')) ||
       ((*p >= '\xD1') && (*p <= '\xDD')))
      *p += 'a' - 'A';

    p++;
    }
  return(s);
}
---------------------------------------------------------------

----------------------------------------------------------------------------
Radio Suisse Ltd,       Internet:  winter@vision.rs.ch
Laupenstrasse 18a                  postmaster@vision.rs.ch
3008 Bern               PSI-Mail: +22846431062::WINTER     (public X.25)
Switzerland             X.400: C=CH; A=ARCOM; P=RS; O=DM; S=WINTER, G=MARTIN
----------------------------------------------------------------------------