Can I haz full unicode support @forums plz? - Page 1
Forum section: electronics / general

author: frozenfrogz
author_rank: Frequent Contributor
timestamp: September 11, 2017, 06:20:50 pm
content: Hello everyone.I did not really know where I should put this, because I could not find a forum feedback section.I am running into issues regarding not supported glyphs on a more or less regular basis. This is no big deal in a general sense, but it annoys the hell out of me. As far as I can see it, the site uses the UTF-8 charset, but the editor allows usage of Unicode characters - these are all correctly displayed. This might be a browser-side thing, but IDK. After posting, all characters not part of UTF-8 get replaced by a question mark.I would love to see either of two options implemented:A: Full support of unicode characters (preferred)B: Disallow everything not UTF-8 compliant in the editor.Is it only me? What is your opinion on that?Best regards,Frederik

author: ataradov
author_rank: Super Contributor
timestamp: September 11, 2017, 07:41:22 pm
content: Editor is browser-side, it will let you enter whatever your OS allows. The forum then goes and filters stuff that is not ASCII.

author: frenky
author_rank: Unknown
timestamp: September 11, 2017, 07:45:31 pm
content: In html of site there is charset=UTF-8 tag so that is ok. Probably DB does not support unicode...Test: ??ž?šShould be:EDIT:Interesting... in preview of the post all characters were ok...

author: frozenfrogz
author_rank: Frequent Contributor
timestamp: September 11, 2017, 08:19:30 pm
content: Quote from: frenky on September 11, 2017, 07:45:31 pmAnd that is the annoying part. Since SOME accents / special chars can be displayed and others can not, it is pretty inconvenient.acute: é á í ú ógrave: è à ì ù òdiaeresis ? ä ï ü öUmlaut: ä ü öcedilla: çscandinavian characters: å æ øThat is as good as it gets, maybe some Slavic characters, but no Russian etc.Edit: Noe_el is out as well xD

author: cdev
author_rank: Super Contributor
timestamp: September 11, 2017, 08:28:32 pm
content: I could see the accented characters..

author: ataradov
author_rank: Super Contributor
timestamp: September 11, 2017, 08:29:44 pm
content: Test of Russian:???.Nope, not at all. And apparently Russian is a lot of confused smiles

author: Zero999
author_rank: Super Contributor
timestamp: September 11, 2017, 08:54:39 pm
content: This has been an issue for as long as the forum has existed and people complain about it, from time to time.Yes, i do find it odd that some non-ASCII characters are supported and not others. I use the clippings extension for Firefox to keep a list of useful non-ASCII characters, which this site supports. Here they are: µ×÷±¼½¾°

author: ataradov
author_rank: Super Contributor
timestamp: September 11, 2017, 08:56:31 pm
content: I think it just supports full extended-ASCII (8-bit): Here is a test of some old-school DOS-style frames: ? ? ?Nope: frames did not go though. So there is some explicit white list.<- Here are Cyrillic leters entered as Alt-{numpad 150, 151, 152}. So it does support simple 8-bit set as input.Nope did not work either. They show up correctly after the posting, but change back to garbage after a page refresh. Strange.

author: Monkeh
author_rank: Super Contributor
timestamp: September 11, 2017, 09:02:44 pm
content: Quote from: ataradov on September 11, 2017, 08:56:31 pmWhich one is that? 8859-1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16? CP-1252?‡ ‰ … €© ªLooks like perhaps CP-1252. The horror.I know conversions often don't go well, but.. please. This sucks.

author: bitseeker
author_rank: Super Contributor
timestamp: September 12, 2017, 12:15:36 am
content: It could be a major overhaul to make the forum truly UTF-8. Limitations are OK if it's clear what they are. That's where the preview functionality could do a better job of delivering an accurate preview prior to posting.At least we can do °C and µV. If onlywas supported as a character, I think that'd cover the "special" characters I generally use in posts here.

author: Bud
author_rank: Super Contributor
timestamp: September 12, 2017, 01:19:27 am
content: Why is this an issue and why is it needed on an English speaking forum? If people will happily post in their favorite languages it will be a mess.

author: ataradov
author_rank: Super Contributor
timestamp: September 12, 2017, 01:22:38 am
content: Quote from: Bud on September 12, 2017, 01:19:27 amWell, the first time I ran into this problem is when I tried to do a translation of a document for a Soviet-made part someone asked about here.There are uses for full language support. It is not critical, but in 2017 is important, IMO.

author: Monkeh
author_rank: Super Contributor
timestamp: September 12, 2017, 01:35:56 am
content: Quote from: Bud on September 12, 2017, 01:19:27 am? (up, down, left, right arrows)¹ ² ³ × ÷ ± (superscript 1, 2, 3, multiply, divide, plus/minus)? (Omega)® © ™ ((R), (C), TM)° (degree)? ¼ ? ½ ? ¾ ? (1/8 1/4 3/8 1/2 5/8 3/4 7/8)Some gets through. Some doesn't. Remembering what does and doesn't work is annoying.E: For giggles,a few more by row:¬!"£$%^&*()_+|¡?£¼???™±°¿??E®?¥??ØÞÆ§Ðª???¦<>©‘’Nº×÷

author: Bud
author_rank: Super Contributor
timestamp: September 12, 2017, 01:40:50 am
content: Honestly, from your list it is not a problem for most of those to use conventional ASCII equivalents and the other ones such as the copyright character may only be needed once in a blue moon.

author: Monkeh
author_rank: Super Contributor
timestamp: September 12, 2017, 01:42:29 am
content: Quote from: Bud on September 12, 2017, 01:40:50 amAnd when I decide to copy and paste something from a document which makes use of these wonderful computers we've developed since the 1960s? I can't tell what's screwed up until it's posted and I can't remember the list of needless limitations.I really like typing µ and ?. Ah, crap, no ?.?·m. N·m.Oh hey, we do get interpunct, that's something.

author: Ampera
author_rank: Super Contributor
timestamp: September 12, 2017, 04:00:12 am
content: Well you can always do

author: bluevd
author_rank: Contributor
timestamp: September 12, 2017, 07:13:24 am
content: The problem is much more complicated that it would look at first sight.Consider the software involved: web server (most likely Apache), script interpreters (PHP), database (MySQL) and dbms (mysqli or whatever php has).And last but not least, the forum software.If you ever tried to get the above combo to support everything you know how much of a pain it can be. Sometimes you go through all the pain to realize the forum scripts strip special characters out.Most likely, you'd be better served by a Pastebin service that supports unicode encoding for posting manual translations and such.

author: Ian.M
author_rank: Super Contributor
timestamp: September 12, 2017, 07:24:28 am
content: Actually v2 of SMF forum does support Unicode *IF* it is enabled. The problem is the message database that is in whatever legacy encoding that was chosen when the forum was originally installed before SMF unicode support. There is an official conversion procedure but I believe there is a considerable risk of data loss so it would be a lot of work for Gnif to activate it then resolve the resulting problems.Seehttps://wiki.simplemachines.org/smf/UTF-8_Readme

author: rs20
author_rank: Super Contributor
timestamp: September 12, 2017, 07:31:52 am
content: Quote from: frozenfrogz on September 11, 2017, 06:20:50 pmUTF-8 is an encoding of Unicode, not an alternative. UTF-8 can encode any Unicode character.Quote from: bluevd on September 12, 2017, 07:13:24 amNope, done it before, it's trivial if you're paying attention. If the mods were to set up a brand new EEVBlog forumfrom scratch, tomorrow, I bet they'd use UTF-8 throughout without any trouble. But what's really difficult, and the semi-legit reason why this change isn't being made, is that themigrationfrom ASCII to UTF-8 is a huge and risky job -- you have to change all these moving parts in parallel, simultaneously, and also modify the encoding in the database and possibly re-encode everything in the database. It's not the sort of thing you can do casually over a few weekends.

author: A Hellene
author_rank: Frequent Contributor
timestamp: September 12, 2017, 08:48:07 am
content: Yet, Unicode support is still there for older messages; for example:https://www.eevblog.com/forum/chat/an-partial-goodbye-could-be/msg42748/#msg42748How can this be?-George

author: tooki
author_rank: Super Contributor
timestamp: September 12, 2017, 08:56:17 am
content: Yes, a switch to Unicode is about 15 years overdue. Aside from scientific notation, there are also things like users asking for translations, or phonetic transcription, just to list some examples. It boggles my mind that anyone in this century would configure a forum as ASCII...

author: wraper
author_rank: Unknown
timestamp: September 12, 2017, 09:08:28 am
content: Quote from: TwoOfFive on September 12, 2017, 04:00:12 amYou can, and it's a big PITA. When people ask to translate something about Soviet components or similar, it's a huge time waste. And the worst part is that preview shows everything correctly. So you post just to find a bunch of question marks and smilies. Also, when you replace text with pictures, it's no longer possible to copy and google it, which often would be very useful. For example, to find Russian documentation, or suggest a search term for taobao.

author: frozenfrogz
author_rank: Frequent Contributor
timestamp: September 12, 2017, 09:38:51 am
content: Quote from: rs20 on September 12, 2017, 07:31:52 amSorry, I got that wrong. So my request would be for full Unicode char set.Since the forum seems to fully support UTF-8 encoding and can display all Unicode characters (?), the editor might be to blame. Maybe it is just a mismatch of the character encoding in the editor vs. the database.

author: A Hellene
author_rank: Frequent Contributor
timestamp: September 12, 2017, 10:03:03 am
content: Quote from: frozenfrogz on September 12, 2017, 09:38:51 amOr, perhaps, some kind offilterbetween the Editor (that seems to be supporting Unicode during preview) and the Database (thatsupports Unicode, as can be seen below)?I do not really know...-George

author: capt bullshot
author_rank: Super Contributor
timestamp: September 12, 2017, 10:03:13 am
content: Bwargh. Stay away with all that charset confusion.Anything else but 7 Bit ASCII is to be considered bad!

author: Zero999
author_rank: Super Contributor
timestamp: September 12, 2017, 10:32:22 am
content: Quote from: Monkeh on September 12, 2017, 01:35:56 amThe forum supports superscript and subscript, so the superscript and fraction characters aren't needed. It also makes the fractions easier to read because they're slightly larger1/2and the letter o in superscript makes a good degree sign 180o.Testing ? greater than or equal to, ? less than or equal to.

author: EEVblog
author_rank: Unknown
timestamp: September 12, 2017, 10:34:15 am
content: SMF has the ability to do this with a single click on a button in the maintenance section (assuming no other issues), the problem is finding issues with exiting content (assuming it "just works" of course).So it would likely have to be done on a copy of the forum first that is available to the public so that the checking of existing content can be crowd sourced, but it's not like there will be thousands who will bother to do this. Even then there could be old posts that get corrupted in some way.

author: EEVblog
author_rank: Unknown
timestamp: September 12, 2017, 10:37:02 am
content: Quote from: Bud on September 12, 2017, 01:19:27 amYes, I do not want people to start posting in other languages.If language is only major reason for wanting to switch to Unicode then I'm afraid the answer is going to be no.

author: Zero999
author_rank: Super Contributor
timestamp: September 12, 2017, 10:40:24 am
content: Quote from: EEVblog on September 12, 2017, 10:37:02 amI agree, but I don't think that is the main reason why people want it. Greek characters, such as the Ohm symbol, are probably the deciding factor. I know there's the Latex math plug-in but it's inconvenient and won't work when scripting is disabled, as is often the case for those using this site behind a firewall.

author: Ampera
author_rank: Super Contributor
timestamp: September 12, 2017, 11:25:54 am
content: Quote from: capt bullshot on September 12, 2017, 10:03:13 amLol, only the best computers use EBCDIC.

author: tooki
author_rank: Super Contributor
timestamp: September 12, 2017, 11:39:59 am
content: Quote from: EEVblog on September 12, 2017, 10:37:02 amNobody is suggesting that there suddenly be threads held in Cherokee and Tibetan!!! Talk about reductio ad absurdum...Supporting foreign characters is highly useful for us English speakers asking help of others in translating texts we cannot read. Google translate can only go so far, you know??

author: tooki
author_rank: Super Contributor
timestamp: September 12, 2017, 11:42:21 am
content: Quote from: EEVblog on September 12, 2017, 10:34:15 amFWIW, converting text from ASCII to Unicode is generally unproblematic, since everything in ASCII has a complete representation in Unicode. Going the other way is a goddamned nightmare, of course!!In fact, modern OSes use Unicode internally anyway, converting most text inputs to Unicode in the system libraries.

author: frozenfrogz
author_rank: Frequent Contributor
timestamp: September 12, 2017, 11:54:05 am
content: Seems like I hit a nerve here...My request is not related to being able to talk in Croatian or Hindi with other people, but I run into the editor barrier quite often.As I stated in the opening post it is more an inconvenience that annoys me from time to time and I see a lot of benefits on the usability side if the displayed in the editor vs. displayed after posting was sorted out one way or the other.Some real world examples of the past couple of weeks I ran into:Writing the Cyrillic name of my "RKSB-104" failed. That way it is pretty hard talking about e.g. Russian schematics.Spelling a polish name correctly failed (I wanted to be obliging).Mathematical equations are very likely to end up as garbled nonsense.This affects forum posts and also private messages.

author: wraper
author_rank: Unknown
timestamp: September 12, 2017, 12:18:12 pm
content: Quote from: EEVblog on September 12, 2017, 10:37:02 amIf this was an issue, then there would be a lot of non English posts already. For example, I don't see problems writing in German, Dutch or many other languages using Latin alphabet.Example: Straße, šprotes.Even if some letters are missing, it's not a big issue for such.

author: Monkeh
author_rank: Super Contributor
timestamp: September 12, 2017, 01:20:25 pm
content: Quote from: Hero999 on September 12, 2017, 10:32:22 amThey're a huge pain in the arse to use. Also, superscript (at least for the values included in the deficient CP1252) works. It has a good set of characters.. for 1985.Again, it's not about what you can bodge in with this editor - if I've written something in a normal fashion using a non-broken charset, I should be able to paste it here and have it display. It's 2017: We should be far, far beyond these daft limitations.Quote from: EEVblog on September 12, 2017, 10:34:15 amAs opposed to all the old, and new, posts whicharecorrupted and will continue to be???Have some eighth ohm corruption.

author: rs20
author_rank: Super Contributor
timestamp: September 12, 2017, 02:08:56 pm
content: Yeah, sorry Dave, I can agree (and indeed was one of the first to put forward in this thread) that the migration might be arduous and perhaps not worth the trouble, but I really disagree that having proper unicode support is actuallyworsethan the status quo. Unicode is far, far more* than just unfamiliar languages, and as already put forward, snippets of other languages are perfectly justified anyway.* Having an ohm symbol that isn't as misaligned as in 5kwould be a great start...

author: A Hellene
author_rank: Frequent Contributor
timestamp: September 12, 2017, 02:16:30 pm
content: Today, even eight bitters support extended character sets...-George

author: A Hellene
author_rank: Frequent Contributor
timestamp: September 12, 2017, 06:13:48 pm
content: By the way, let me translate what the 84x48 dot-matrix LCD above says, word-by-word:At the 'Runtime.jpg':-Line 1: 12.20 V-Line 2: 64.8 m?-Line 3: 135.8 A...meaning:-Line 1: 12.20 V-Line 2: 64.8 mohm (m+Omega)-Line 3: 135.8 AAnd at the 'Runtime.jpg':-Line 1:-Line 2:??-Line 3:??.-Line 4:: 1.05-Line 5:??-Line 6: 22?. 2017...loosely meaning: Lead(-Acid) accumulator (=battery) Desulfator-Line 1: Desulfator-Line 2: accumulators-Line 3:(of) Lead(-Acid).-Line 4: Version: 1.05-Line 5: GeorgeKolovos (no White Space, due to LCD width limit)-Line 6: Jul. 22, 2017Makes any sense?This is what I have posted:-George