Redmine - Defect #24616
Should not replace all invalid utf8 characters (e.g in mail)2016-12-15 02:13 - Pavel Rosický

ステータス:

Closed

開始日:

優先度:

Normal

期日:

担当者:
カテゴリ:

I18n

対象バージョン:3.4.0

Resolution:

Fixed

進捗率:

0%

予定工数:

0.00時間Affected version:

説明
Hello,
I've an email, that is encoded in utf8, but it contains an invalid character. In this case, redmine converts the content to us-ascii andthen to utf8. This step will replace non-ascii compatible chars to "?". Why?
1) Failure:
MailHandlerTest#test̲invalid̲utf8 [/test/unit/mail̲handler̲test.rb:548]:Expected: "Здравствуйте?"
Actual: "?????????????"
I changed Redmine::CodesetUtil.replace̲invalid̲utf8(str) and Redmine::CodesetUtil.to̲utf8(str, encoding)str=str.encode("US-ASCII",

str=str.encode("UTF-8",:invalid=>:replace,:undef=>:replace,:replace=>'?').encode("UTF-8")

to:invalid=>:replace,:undef=>:replace,:replace=>'?')

all tests are passing with this change.
Redmine version
Ruby version
Rails version
Environment
Database adapter
SCM:
GitFilesystem
Redmine plugins:no plugin installed

3.3.1.stable
2.1.5-p273 (2014-11-13)
4.2.7.1
production
Mysql2[x64-mingw32]

2.10.1

関係しているリビジョン
リビジョン 16271 - 2017-01-28 06:43 - Toshi MARUYAMA
add more tests (#24616)
リビジョン 16272 - 2017-01-28 06:43 - Toshi MARUYAMA
unify duplicate codes (#24616)
リビジョン 16273 - 2017-01-28 07:24 - Toshi MARUYAMA
do not replace all invalid utf8 (#24616)
リビジョン 16274 - 2017-01-28 07:25 - Toshi MARUYAMA
additional test for mail by Pavel Rosický (#24616)
リビジョン 16275 - 2017-01-28 07:29 - Toshi MARUYAMA
svn propset svn:eol-style native test/fixtures/mail̲handler/invalid̲utf8.eml (#24616)

2024-09-16

1/3

履歴
#1 - 2016-12-29 08:14 - Go MAEDA- ファイル defect-24616.diff を追加- 対象バージョン を 3.3.2 にセット
Looks good to me.
# valid UTF-8 string
text = "こんにちは"
p text.valid̲encoding? #=>true# making invalid UTF-8 string
text.force̲encoding('ASCII-8BIT')
text[-1] = 0xff.chr
text.force̲encoding("UTF-8")
p text.valid̲encoding? # => false
p text# => "こんにち\xE3\x81\xFF"# Current code of Redmine
p text.encode("US-ASCII", :invalid => :replace, :undef => :replace, :replace => '?').encode("UTF-8")# => "??????"# Fixed code by Pavel Rosický
p text.encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => '?')
# => "こんにち??"#2 - 2016-12-29 09:31 - Toshi MARUYAMA- 対象バージョン を削除 (3.3.2)
Did you run whole tests?
Especially this test.
source:tags/3.3.1/test/unit/lib/redmine/codeset̲util̲test.rb#3 - 2016-12-29 09:34 - Toshi MARUYAMA
Pavel Rosický wrote:
Hello,I've an email, that is encoded in utf8, but it contains an invalid character. In this case, redmine converts the content to us-ascii and then to
utf8. This step will replace non-ascii compatible chars to "?". Why?

You can see this function purpose.
source:tags/3.3.1/test/unit/lib/redmine/codeset̲util̲test.rb#L68#4 - 2016-12-29 10:41 - Pavel Rosický
Thanks Toshi, I rechecked it again and all tests are passing.
source:tags/3.3.1/test/unit/lib/redmine/codeset̲util̲test.rb#L68
In this case, my change has no effect on the result, because the string contains just one invalid utf-8 character.
s1.encode('us-ascii', :invalid => :replace,"Texte encod? en ISO-8859-1."# patcheds1.encode('utf-8', :invalid => :replace,"Texte encod? en ISO-8859-1.":undef:undef=>=>:replace,:replace,:replace:replace=>=>'?').encode('utf-8')'?')

but a combination of valid and invalid utf-8 chars (non-ascii-compatible) will result both characters are stripped. Try out GO Media's example.
#5 - 2017-01-28 05:19 - Toshi MARUYAMA$ irb
1.9.3-p551 :001 > text = "こんにち\xE3\x81\xFF"=> "こんにち\xE3\x81\xFF"1.9.3-p551 :002 > text =text.encode("UTF-8", :invalid=> "こんにち\xE3\x81\xFF"1.9.3-p551 :003 > text.valid̲encoding?
=> false

2024-09-16=>:replace,:undef=>:replace,:replace=>'?')

2/3

$ irb
2.3.3 :001 > text = "こんにち\xE3\x81\xFF"=> "こんにち\xE3\x81\xFF"2.3.3 :002 > text =text.encode("UTF-8", :invalid=> "こんにち??"
2.3.3 :003 > text.valid̲encoding?
=> true=>:replace,:undef=>:replace,:replace=>'?')#6 - 2017-01-28 07:07 - Toshi MARUYAMAPavel Rosický wrote:
Hello,I've an email, that is encoded in utf8, but it contains an invalid character. In this case, redmine converts the content to us-ascii and then to
utf8. This step will replace non-ascii compatible chars to "?". Why?

Because of Ruby 1.8.7 behavior compatibility.
source:tags/2.6.9/lib/redmine/codeset̲util.rb#7 - 2017-01-28 07:09 - Toshi MARUYAMA- 題名 を encoding error if email contains an invalid utf8 character から Should not replace all invalid utf8 characters に変更- カテゴリ を Email receiving から I18n に変更#8 - 2017-01-28 07:10 - Toshi MARUYAMA- 題名 を Should not replace all invalid utf8 characters から Should not replace all invalid utf8 characters (e.g in mail) に変更#9 - 2017-01-28 08:56 - Toshi MARUYAMA- ステータス を New から Closed に変更- 対象バージョン を 3.4.0 にセット
- Resolution を Fixed にセットI have committed r16273 to pass on Ruby 1.9.3.
I don't want to change behavior on stable.

ファイル
invalid̲utf8̲test.patch

1.49 KB

2016-12-15

Pavel Rosický

defect-24616.diff

1.73 KB

2016-12-29

Go MAEDA

2024-09-16

3/3