Repository: flavorjones/loofah (Ruby)

Issue:
`#to_text` doesn't handle `<br>` elements well.
`#to_text` doesn't handle `<br>` elements explicitly, so phrases like `"foo<br>bar"` get sanitzed to `"foobar"`. In our use-case a line break (`"foo\nbar"`) is a more practical result.

So I wrote a custom scrubber:
```ruby
class HTMLToPlainText
  class BreakElementScrubber < Loofah::Scrubber
    def scrub(node)
      return CONTINUE unless node.name == 'br'
 
      node.add_previous_sibling Nokogiri::XML::Text.new("\n", node.document)
      node.remove
    end
  end

  def call(html_string)
    Loofah::HTML::Document
      .parse(html_string)
      .scrub!(:prune)
      .scrub!(BreakElementScrubber.new)
      .to_text(encode_special_chars: false)
      .strip
  end
end
```

Is this something valuable which should find it's way in the project via a pull request? If yes, could a maintainer give me implementation hints etc.? I'm willing to help.


Discussion:
Hi, @der-flo! Thanks for pointing this out, that seems like a really good feature to add. I don't think we need a new scrubber for it, though, I think we should modify the current behavior of `#to_text` (or, really, the behavior of the underlying scrubber, `Loofah::Scrubbers::NewlineBlockElements`.

That scrubber currently adds newlines for "block" elements in HTML4 and HTML5 plus a loose set of elements that can contain block elements. But it doesn't pay attention to the element `br` which is the _direct_ analog of a newline in HTML.

I'll write up a PR for this now, it should be a very small change.
Loofah 2.14.0 has been released with `#to_text` inserting newlines for `<br>` elements. I hope this meets your needs!
Your solution works fine, thanks!
Happy to help!



Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 68dce47..12b4059 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,12 @@
 # Changelog
 
+## next / unreleased
+
+### Features
+
+* The `#to_text` method on `Loofah::HTML::{Document,DocumentFragment}` replaces `<br>` linebreak elements with a newline. [[#225](https://github.com/flavorjones/loofah/issues/225)]
+
+
 ## 2.13.0 / 2021-12-10
 
 ### Bug fixes
diff --git a/README.md b/README.md
index a27dbe8..3de845d 100644
--- a/README.md
+++ b/README.md
@@ -133,13 +133,12 @@ and `text` to return plain text:
 doc.text    # => "ohai! div is safe "
 ```
 
-Also, `to_text` is available, which does the right thing with
-whitespace around block-level elements.
+Also, `to_text` is available, which does the right thing with whitespace around block-level and line break elements.
 
 ``` ruby
-doc = Loofah.fragment("<h1>Title</h1><div>Content</div>")
-doc.text    # => "TitleContent"           # probably not what you want
-doc.to_text # => "\nTitle\n\nContent\n"   # better
+doc = Loofah.fragment("<h1>Title</h1><div>Content<br>Next line</div>")
+doc.text    # => "TitleContentNext line"            # probably not what you want
+doc.to_text # => "\nTitle\n\nContent\nNext line\n"  # better
 ```
 
 ### Loofah::XML::Document and Loofah::XML::DocumentFragment
diff --git a/lib/loofah/elements.rb b/lib/loofah/elements.rb
index d2ae4bf..5596e6f 100644
--- a/lib/loofah/elements.rb
+++ b/lib/loofah/elements.rb
@@ -70,8 +70,6 @@ module Elements
                                          video
                                        ]
 
-    STRICT_BLOCK_LEVEL = STRICT_BLOCK_LEVEL_HTML4 + STRICT_BLOCK_LEVEL_HTML5
-
     # The following elements may also be considered block-level
     # elements since they may contain block-level elements
     LOOSE_BLOCK_LEVEL = Set.new %w[dd
@@ -86,7 +84,12 @@ module Elements
                                    tr
                                 ]
 
+    # Elements that aren't block but should generate a newline in #to_text
+    INLINE_LINE_BREAK = Set.new(["br"])
+
+    STRICT_BLOCK_LEVEL = STRICT_BLOCK_LEVEL_HTML4 + STRICT_BLOCK_LEVEL_HTML5
     BLOCK_LEVEL = STRICT_BLOCK_LEVEL + LOOSE_BLOCK_LEVEL
+    LINEBREAKERS = BLOCK_LEVEL + INLINE_LINE_BREAK
   end
 
   ::Loofah::MetaHelpers.add_downcased_set_members_to_all_set_constants ::Loofah::Elements
diff --git a/lib/loofah/instance_methods.rb b/lib/loofah/instance_methods.rb
index 316ac9d..7200504 100644
--- a/lib/loofah/instance_methods.rb
+++ b/lib/loofah/instance_methods.rb
@@ -112,11 +112,11 @@ def text(options = {})
     #  Returns a plain-text version of the markup contained by the
     #  fragment, with HTML entities encoded.
     #
-    #  This method is slower than #to_text, but is clever about
-    #  whitespace around block elements.
+    #  This method is slower than #text, but is clever about
+    #  whitespace around block elements and line break elements.
     #
-    #    Loofah.document("<h1>Title</h1><div>Content</div>").to_text
-    #    # => "\nTitle\n\nContent\n"
+    #    Loofah.document("<h1>Title</h1><div>Content<br>Next line</div>").to_text
+    #    # => "\nTitle\n\nContent\nNext line\n"
     #
     def to_text(options = {})
       Loofah.remove_extraneous_whitespace self.dup.scrub!(:newline_block_elements).text(options)
diff --git a/lib/loofah/scrubbers.rb b/lib/loofah/scrubbers.rb
index 14b9786..a0093e9 100644
--- a/lib/loofah/scrubbers.rb
+++ b/lib/loofah/scrubbers.rb
@@ -240,8 +240,13 @@ def initialize
       end
 
       def scrub(node)
-        return CONTINUE unless Loofah::Elements::BLOCK_LEVEL.include?(node.name)
-        node.add_next_sibling Nokogiri::XML::Text.new("\n#{node.content}\n", node.document)
+        return CONTINUE unless Loofah::Elements::LINEBREAKERS.include?(node.name)
+        replacement = if Loofah::Elements::INLINE_LINE_BREAK.include?(node.name)
+          "\n"
+        else
+          "\n#{node.content}\n"
+        end
+        node.add_next_sibling Nokogiri::XML::Text.new(replacement, node.document)
         node.remove
       end
     end


Test patch:
diff --git a/test/integration/test_html.rb b/test/integration/test_html.rb
index 8ae60a8..9db171a 100644
--- a/test/integration/test_html.rb
+++ b/test/integration/test_html.rb
@@ -51,6 +51,11 @@ class IntegrationTestHtml < Loofah::TestCase
         html = Loofah.fragment "<div>tweedle\n\n\t\n\s\nbeetle</div>"
         assert_equal "\ntweedle\n\nbeetle\n", html.to_text
       end
+
+      it "replaces <br> with newlines" do
+        html = Loofah.fragment("hello<div>first line<br>second line</div>goodbye")
+        assert_equal("hello\nfirst line\nsecond line\ngoodbye", html.to_text)
+      end
     end
 
     context "with an `encoding` arg" do
@@ -84,6 +89,11 @@ class IntegrationTestHtml < Loofah::TestCase
         html = Loofah.document "<div>tweedle\n\n\t\n\s\nbeetle</div>"
         assert_equal "\ntweedle\n\nbeetle\n", html.to_text
       end
+
+      it "replaces <br> with newlines" do
+        html = Loofah.document("<body>hello<div>first line<br>second line</div>goodbye</body>")
+        assert_equal("hello\nfirst line\nsecond line\ngoodbye", html.to_text)
+      end
     end
   end
 end