Repository: biolab/orange3-text

Issue:
Question: How to import multiple files or multiple directories?
Hello,
  Nice to see that we got a very fresh version.

  Question: Is it possible to load multiple files and multiple directories for Text Mining?
  I tried opening the whole level up, no luck
  I tried compressing on a file.tsv.gz using tar, no luck

  As it is a group of directories with multiple files (one per document) it would be normal to create a Corpus from such structure. If not, what would work?

  I have something like:

```
Groups
+- Group_1
|  +- 1999
|  |  +- file_1.txt (any valid extension)
|  |  +- . . .
|  +- 2000
|  +- . . .
+- Group_2
    +- 1982
    +- 1985
    +- . . .
```

Thanks and Regards



Discussion:
@angeloklin unfortunately we do not support this yet. This is something we were thinking about on several occasions. Until this is implemented, I don't think there is any other way than concatenating them manually into one of the Orange's supported files (csv, tab, ...). 

Interim, you could construct a single CSV of all documents:

``` text
group_id,year,filename,contents
group_1,1999,file1.txt,"whole file1 contents"
group_1,1999,file2.txt,"whole file2 contents"
...
group_n,2025,fileF.txt,"whole fileF contents"
```

You would have to ensure the quotes in whole file contents are properly escaped.

@nikicc, @kernc Thanks for your reply.
Concatenating is not ideal, as there are 25,000 docs of variable size, on a total of 3.5Gb.

Two other ideas:
1) Would be possible to write a Python script and build a Data Table?
2) I saw the SQL Table widget, but could not identify which databases it can talk with. Options that come to mind in the open surce would be mysql, postgre and mongo, or even simpler sqlite3.

Update: I later checked and found support for postgre via a python library. I wonder if is just a matter of having a library and what other steps to make a db2 available.

Regards 

> Would be possible to write a Python script and build a Data Table?

Yes, that's possible. This is far from ideal but here is a simple example on how to achieve this in the current version. Open the Python Script widget and paste this code:

``` py
import os
import numpy as np
from Orange.data.domain import Domain, StringVariable
from orangecontrib.text.corpus import Corpus


PATH = '<the-path-to-the-group-folder>'
MAX_DOC = 1000

# create Orange's domain; all features will be in metas
meta_attrs = [StringVariable('group'), StringVariable('year'), StringVariable('filename'), StringVariable('content')]
domain = Domain(attributes=[], class_vars=[], metas=meta_attrs)

# create numpy array for storing the data
data = np.empty((MAX_DOC, 4), dtype=object)
doc_ind = 0

class BreakIt(Exception): pass    # enable stopping
try:
    for dir, _, files in os.walk(PATH):                # walk through all sub-directories
        for file in files:                             # and through all files inside sub-directories
            if file.endswith('.txt'):                  # if the type matches
                group, year = dir.split(os.sep)[-2:]   # parse group and year

                # read file's content and delete \n characters
                with open(os.path.join(dir, file), 'r') as f:
                    content = f.read().replace('\n', ' ')

                # store the data to numpy matrix    
                data[doc_ind, 0] = group
                data[doc_ind, 1] = year
                data[doc_ind, 2] = file
                data[doc_ind, 3] = content
                doc_ind += 1

                if doc_ind == MAX_DOC:    # if you loaded enough, stop
                    raise BreakIt
except BreakIt:
    pass

# truncate empty lines
data = data[:doc_ind, :]
print('Loaded {} documents.'.format(doc_ind))

# create corpus
corp = Corpus.from_numpy(domain, X=np.empty((doc_ind, 0)), metas=data)
# set which features to use for text mining; use only content
corp.set_text_features(meta_attrs[-1:])

# send the data to the output
out_object = corp
```

What this does is walks through the folder hierarchy and loads all txt files. All you have to set is the `PATH` variable at the top of the script, which should point to the root `groups` folder. Note that there is also a `MAX_DOC` variable that stops loading after `MAX_DOC` number of documents have been loaded.

Once you load the documents using this code a Corpus will be present on the output signal `out_object` of the widget. You can connect it to other text widgets as you would normally do, just make sure that you are using `out_object` and not `out_data` (this could be changed by clicking on the line between the two widgets).

I should also warn you that you might encounter some memory issues when trying to load 3.5 GB data set into Orange. I suggest you start with some small subset of documents. Also let us know, how this will go.

> Update: I later checked and found support for postgre via a python library. I wonder if is just a matter of having a library and what other steps to make a db2 available.

I don't know much about SQL support. @lanzagar can you help?

@nikicc @kernc  Please suggest if there is a widget in latest version(using 3.4).

@bandaruvenky unfortunately, there isn't yet.

Once it's made, this issue will be closed, so you should be notified about this. 

Resolving patch:
diff --git a/orangecontrib/text/import_documents.py b/orangecontrib/text/import_documents.py
new file mode 100644
--- /dev/null
+++ b/orangecontrib/text/import_documents.py
@@ -0,0 +1,262 @@
+import fnmatch
+import logging
+import os
+import pathlib
+import re
+
+from collections import namedtuple
+from types import SimpleNamespace as namespace
+
+import numpy as np
+
+import docx2txt
+from odf.opendocument import load
+from odf import text, teletype
+
+from pdfminer.pdfparser import PDFParser, PDFDocument
+from pdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreter
+from pdfminer.converter import PDFPageAggregator
+from pdfminer.layout import LAParams, LTTextBox, LTTextLine
+from bs4 import BeautifulSoup
+
+from Orange.data import DiscreteVariable, Domain, StringVariable
+from Orange.data.io import detect_encoding
+from Orange.util import Registry
+
+from orangecontrib.text.corpus import Corpus
+
+
+DefaultFormats = ("docx", "odt", "txt", "pdf", "xml")
+
+TextData = namedtuple(
+    "Text",
+    ["name", "path", "ext", "category", "content"]
+)
+TextData.isvalid = property(lambda self: True)
+
+TextDataError = namedtuple(
+    "TextDataError",
+    ["path", "error", "error_str"]
+)
+TextDataError.isvalid = property(lambda self: False)
+
+log = logging.getLogger(__name__)
+
+
+class Reader(metaclass=Registry):
+    def __init__(self, path, replace_white_space=False):
+        self.path = path
+        self.replace_white_space = replace_white_space
+        self.content = None
+
+    @classmethod
+    def get_reader(cls, path):
+        ext = pathlib.Path(path).suffix
+        for _reader in cls.registry:
+            reader = eval(_reader)
+            if ext in reader.ext:
+                return reader(path)
+        return Reader(path)
+
+    def read(self, ):
+        error = ""
+        try:
+            self.read_file()
+        except Exception as ex:
+            textdata = None
+            error = "{}".format(pathlib.Path(self.path).name)
+            log.exception('Error reading failed', exc_info=ex)
+        else:
+            textdata = self.make_text_data()
+        return textdata, error
+
+    def read_file(self):
+        raise NotImplementedError("No reader for {}".format(pathlib.Path(self.path).suffix))
+
+    def make_text_data(self):
+        name = pathlib.Path(self.path).stem
+        directory = pathlib.PurePath(self.path).parent
+        category = directory.parts[-1] or "None"
+        if self.replace_white_space:
+            self.content = re.sub('\s+', ' ', self.content)
+        return TextData(name, self.path, self.ext, category, self.content)
+
+
+class TxtReader(Reader):
+    ext = [".txt"]
+
+    def read_file(self):
+        encoding = detect_encoding(self.path)
+        with open(self.path, 'r', encoding=encoding) as f:
+            self.content = f.read()
+
+
+class DocxReader(Reader):
+    ext = [".docx"]
+
+    def read_file(self):
+        self.content = docx2txt.process(self.path)
+
+
+class OdtReader(Reader):
+    ext = [".odt"]
+
+    def read_file(self):
+        odtfile = load(self.path)
+        texts = odtfile.getElementsByType(text.P)
+        self.content = " ".join(teletype.extractText(t) for t in texts)
+
+
+class PdfReader(Reader):
+    """
+    char_margin — two text chunks whose distance is closer than this value are considered
+    contiguous and get grouped into one.
+    word_margin — it may be required to insert blank characters (spaces) as necessary if
+    the distance between two words is greater than this value, as a blank between words might
+    not be represented as a space, but indicated by the positioning of each word.
+    """
+    ext = [".pdf"]
+
+    def read_file(self):
+        with open(self.path, 'rb') as f:
+            parser = PDFParser(f)
+        doc = PDFDocument()
+        parser.set_document(doc)
+        doc.set_parser(parser)
+        doc.initialize('')
+        rsrcmgr = PDFResourceManager()
+        laparams = LAParams()
+        laparams.char_margin = 0.1
+        laparams.word_margin = 1.0
+        device = PDFPageAggregator(rsrcmgr, laparams=laparams)
+        interpreter = PDFPageInterpreter(rsrcmgr, device)
+        extracted_text = []
+
+        for page in doc.get_pages():
+            interpreter.process_page(page)
+            layout = device.get_result()
+            for lt_obj in layout:
+                if isinstance(lt_obj, LTTextBox) or isinstance(lt_obj, LTTextLine):
+                    extracted_text.append(lt_obj.get_text())
+        self.content = ' '.join(extracted_text)
+
+
+class XmlReader(Reader):
+    ext = [".xml"]
+
+    def read_file(self):
+        encoding = detect_encoding(self.path)
+        with open(self.path, encoding=encoding, errors='ignore') as markup:
+            soup = BeautifulSoup(markup.read(), "lxml")
+        self.content = soup.get_text()
+
+
+class ImportDocuments:
+    def __init__(self, startdir, formats=DefaultFormats, report_progress=None):
+        self.startdir = startdir
+        self.formats = formats
+        self._report_progress = report_progress
+        self.cancelled = False
+        self._text_data = []
+
+    def run(self):
+        text_data = []
+        errors = []
+        patterns = ["*.{}".format(fmt.lower()) for fmt in self.formats]
+        paths = self.scan(self.startdir, include_patterns=patterns)
+        n_paths = len(paths)
+        batch = []
+
+        for path in paths:
+            if len(batch) == 1 and self._report_progress is not None:
+                self._report_progress(
+                    namespace(progress=len(text_data) / n_paths,
+                              lastpath=path,
+                              batch=batch))
+                batch = []
+
+            reader = Reader.get_reader(path)
+            text, error = reader.read()
+            if text is not None:
+                text_data.append(text)
+                batch.append(text_data)
+            else:
+                errors.append(error)
+
+            if self.cancelled:
+                return
+
+        self._text_data = text_data
+        return self._create_corpus(), errors
+
+    def _create_corpus(self):
+        corpus = None
+        names = ["name", "path", "content"]
+        data = []
+        category_data = []
+        text_categories = list(set(t.category for t in self._text_data))
+        values = list(set(text_categories))
+        category_var = DiscreteVariable.make("category", values=values)
+        for textdata in self._text_data:
+            data.append(
+                [textdata.name,
+                 textdata.path,
+                 textdata.content]
+            )
+            category_data.append(category_var.to_val(textdata.category))
+        if len(text_categories) > 1:
+            category_data = np.array(category_data)
+        else:
+            category_var = []
+            category_data = np.empty((len(data), 0))
+        domain = Domain(
+            [], category_var, [StringVariable.make(name) for name in names]
+        )
+        domain["name"].attributes["title"] = True
+        data = np.array(data, dtype=object)
+        if len(data):
+            corpus = Corpus(domain,
+                            Y=category_data,
+                            metas=data,
+                            text_features=[domain.metas[2]])
+
+        return corpus
+
+    @staticmethod
+    def scan(topdir, include_patterns=("*",), exclude_patterns=(".*",)):
+        """
+        Yield file system paths under `topdir` that match include/exclude patterns
+
+        Parameters
+        ----------
+        topdir: str
+            Top level directory path for the search.
+        include_patterns: List[str]
+            `fnmatch.fnmatch` include patterns.
+        exclude_patterns: List[str]
+            `fnmatch.fnmatch` exclude patterns.
+
+        Returns
+        -------
+        list of paths
+        """
+        if include_patterns is None:
+            include_patterns = ["*"]
+
+        def matches_any(fname, patterns):
+            return any(fnmatch.fnmatch(fname.lower(), pattern)
+                       for pattern in patterns)
+
+        paths = []
+
+        for dirpath, dirnames, filenames in os.walk(topdir):
+            for dirname in list(dirnames):
+                # do not recurse into hidden dirs
+                if fnmatch.fnmatch(dirname, ".*"):
+                    dirnames.remove(dirname)
+
+            filenames = [fname for fname in filenames
+                         if matches_any(fname, include_patterns)
+                            and not matches_any(fname, exclude_patterns)]
+            paths = paths + [os.path.join(dirpath, fname) for fname in filenames]
+        return paths
diff --git a/orangecontrib/text/widgets/owimportdocuments.py b/orangecontrib/text/widgets/owimportdocuments.py
new file mode 100644
--- /dev/null
+++ b/orangecontrib/text/widgets/owimportdocuments.py
@@ -0,0 +1,635 @@
+"""
+Import Documents
+-------------
+
+Import documents 'into' canvas session from a local file system
+"""
+import sys
+import os
+import enum
+import warnings
+import logging
+import traceback
+
+from types import SimpleNamespace as namespace
+from concurrent.futures._base import TimeoutError
+
+from AnyQt.QtCore import Qt, QEvent, QFileInfo, QThread
+from AnyQt.QtCore import pyqtSlot as Slot
+from AnyQt.QtGui import QStandardItem, QDropEvent
+
+from AnyQt.QtWidgets import (
+    QAction, QPushButton, QComboBox, QApplication, QStyle, QFileDialog,
+    QFileIconProvider, QStackedWidget, QProgressBar, QWidget, QHBoxLayout,
+    QVBoxLayout, QLabel
+)
+
+from Orange.widgets import widget, gui, settings
+from Orange.widgets.utils.filedialogs import RecentPath
+from Orange.widgets.utils.concurrent import (
+    ThreadExecutor, FutureWatcher, methodinvoke
+)
+from Orange.widgets.widget import Output
+
+from orangecontrib.text.corpus import Corpus
+from orangecontrib.text.import_documents import ImportDocuments
+from Orange.canvas.preview.previewbrowser import TextLabel
+
+
+def prettifypath(path):
+    home = os.path.expanduser("~/")
+    if path.startswith(home):  # case sensitivity!
+        path = os.path.join("~", os.path.relpath(path, home))
+    return path
+
+
+log = logging.getLogger(__name__)
+
+
+class RuntimeEvent(QEvent):
+    Init = QEvent.registerEventType()
+
+
+def RecentPath_asqstandarditem(pathitem):
+    icon_provider = QFileIconProvider()
+    # basename of a normalized name (strip right path component separators)
+    basename = os.path.basename(os.path.normpath(pathitem.abspath))
+    item = QStandardItem(
+        icon_provider.icon(QFileInfo(pathitem.abspath)),
+        basename
+    )
+    item.setToolTip(pathitem.abspath)
+    item.setData(pathitem, Qt.UserRole)
+    return item
+
+
+class State(enum.IntEnum):
+    NoState, Processing, Done, Cancelled, Error = range(5)
+
+
+class OWImportDocuments(widget.OWWidget):
+    name = "Import Documents"
+    description = "Import text documents from folders."
+    icon = "icons/ImportDocuments.svg"
+    priority = 11
+
+    class Outputs:
+        data = Output("Corpus", Corpus)
+
+    #: list of recent paths
+    recent_paths = settings.Setting([])  # type: List[RecentPath]
+    currentPath = settings.Setting(None)
+
+    want_main_area = False
+    resizing_enabled = False
+
+    Modality = Qt.ApplicationModal
+
+    MaxRecentItems = 20
+
+
+    class Warning(widget.OWWidget.Warning):
+        read_error = widget.Msg("{} couldn't be read.")
+
+
+    def __init__(self):
+        super().__init__()
+        #: widget's runtime state
+        self.__state = State.NoState
+        self.corpus = None
+        self.n_text_categories = 0
+        self.n_text_data = 0
+        self.n_skipped = 0
+
+        self.__invalidated = False
+        self.__pendingTask = None
+
+        vbox = gui.vBox(self.controlArea)
+        hbox = gui.hBox(vbox)
+        self.recent_cb = QComboBox(
+            sizeAdjustPolicy=QComboBox.AdjustToMinimumContentsLengthWithIcon,
+            minimumContentsLength=16,
+            acceptDrops=True
+        )
+        self.recent_cb.installEventFilter(self)
+        self.recent_cb.activated[int].connect(self.__onRecentActivated)
+
+        browseaction = QAction(
+            "Open/Load Documents", self,
+            iconText="\N{HORIZONTAL ELLIPSIS}",
+            icon=self.style().standardIcon(QStyle.SP_DirOpenIcon),
+            toolTip="Select a folder from which to load the documents"
+        )
+        browseaction.triggered.connect(self.__runOpenDialog)
+        reloadaction = QAction(
+            "Reload", self,
+            icon=self.style().standardIcon(QStyle.SP_BrowserReload),
+            toolTip="Reload current document set"
+        )
+        reloadaction.triggered.connect(self.reload)
+        self.__actions = namespace(
+            browse=browseaction,
+            reload=reloadaction,
+        )
+
+        browsebutton = QPushButton(
+            browseaction.iconText(),
+            icon=browseaction.icon(),
+            toolTip=browseaction.toolTip(),
+            clicked=browseaction.trigger
+        )
+        reloadbutton = QPushButton(
+            reloadaction.iconText(),
+            icon=reloadaction.icon(),
+            clicked=reloadaction.trigger,
+            default=True,
+        )
+
+        hbox.layout().addWidget(self.recent_cb)
+        hbox.layout().addWidget(browsebutton)
+        hbox.layout().addWidget(reloadbutton)
+
+        self.addActions([browseaction, reloadaction])
+
+        reloadaction.changed.connect(
+            lambda: reloadbutton.setEnabled(reloadaction.isEnabled())
+        )
+        box = gui.vBox(vbox, "Info")
+        self.infostack = QStackedWidget()
+
+        self.info_area = QLabel(
+            text="No document set selected",
+            wordWrap=True
+        )
+        self.progress_widget = QProgressBar(
+            minimum=0, maximum=100
+        )
+        self.cancel_button = QPushButton(
+            "Cancel", icon=self.style().standardIcon(QStyle.SP_DialogCancelButton),
+        )
+        self.cancel_button.clicked.connect(self.cancel)
+
+        w = QWidget()
+        vlayout = QVBoxLayout()
+        vlayout.setContentsMargins(0, 0, 0, 0)
+        hlayout = QHBoxLayout()
+        hlayout.setContentsMargins(0, 0, 0, 0)
+
+        hlayout.addWidget(self.progress_widget)
+        hlayout.addWidget(self.cancel_button)
+        vlayout.addLayout(hlayout)
+
+        self.pathlabel = TextLabel()
+        self.pathlabel.setTextElideMode(Qt.ElideMiddle)
+        self.pathlabel.setAttribute(Qt.WA_MacSmallSize)
+
+        vlayout.addWidget(self.pathlabel)
+        w.setLayout(vlayout)
+
+        self.infostack.addWidget(self.info_area)
+        self.infostack.addWidget(w)
+
+        box.layout().addWidget(self.infostack)
+
+        self.__initRecentItemsModel()
+        self.__invalidated = True
+        self.__executor = ThreadExecutor(self)
+
+        QApplication.postEvent(self, QEvent(RuntimeEvent.Init))
+
+    def __initRecentItemsModel(self):
+        if self.currentPath is not None and \
+                not os.path.isdir(self.currentPath):
+            self.currentPath = None
+
+        recent_paths = []
+        for item in self.recent_paths:
+            if os.path.isdir(item.abspath):
+                recent_paths.append(item)
+        recent_paths = recent_paths[:OWImportDocuments.MaxRecentItems]
+        recent_model = self.recent_cb.model()
+        for pathitem in recent_paths:
+            item = RecentPath_asqstandarditem(pathitem)
+            recent_model.appendRow(item)
+
+        self.recent_paths = recent_paths
+
+        if self.currentPath is not None and \
+                os.path.isdir(self.currentPath) and self.recent_paths and \
+                os.path.samefile(self.currentPath, self.recent_paths[0].abspath):
+            self.recent_cb.setCurrentIndex(0)
+        else:
+            self.currentPath = None
+            self.recent_cb.setCurrentIndex(-1)
+        self.__actions.reload.setEnabled(self.currentPath is not None)
+
+    def customEvent(self, event):
+        """Reimplemented."""
+        if event.type() == RuntimeEvent.Init:
+            if self.__invalidated:
+                try:
+                    self.start()
+                finally:
+                    self.__invalidated = False
+
+        super().customEvent(event)
+
+    def __runOpenDialog(self):
+        startdir = os.path.expanduser("~/")
+        if self.recent_paths:
+            startdir = os.path.dirname(self.recent_paths[0].abspath)
+
+        caption = "Select Top Level Folder"
+        if OWImportDocuments.Modality == Qt.WindowModal:
+            dlg = QFileDialog(
+                self, caption, startdir,
+                acceptMode=QFileDialog.AcceptOpen,
+                modal=True,
+            )
+            dlg.setFileMode(QFileDialog.Directory)
+            dlg.setOption(QFileDialog.ShowDirsOnly)
+            dlg.setDirectory(startdir)
+            dlg.setAttribute(Qt.WA_DeleteOnClose)
+
+            @dlg.accepted.connect
+            def on_accepted():
+                dirpath = dlg.selectedFiles()
+                if dirpath:
+                    self.setCurrentPath(dirpath[0])
+                    self.start()
+            dlg.open()
+        else:
+            dirpath = QFileDialog.getExistingDirectory(
+                self, caption, startdir
+            )
+            if dirpath:
+                self.setCurrentPath(dirpath)
+                self.start()
+
+    def __onRecentActivated(self, index):
+        item = self.recent_cb.itemData(index)
+        if item is None:
+            return
+        assert isinstance(item, RecentPath)
+        self.setCurrentPath(item.abspath)
+        self.start()
+
+    def __updateInfo(self):
+        if self.__state == State.NoState:
+            text = "No document set selected"
+        elif self.__state == State.Processing:
+            text = "Processing"
+        elif self.__state == State.Done:
+            nvalid = self.n_text_data
+            ncategories = self.n_text_categories
+            n_skipped = self.n_skipped
+            if ncategories < 2:
+                text = "{} document{}".format(nvalid, "s" if nvalid != 1 else "")
+            else:
+                text = "{} documents / {} categories".format(nvalid, ncategories)
+            if n_skipped > 0:
+                text = text + ", {} skipped".format(n_skipped)
+        elif self.__state == State.Cancelled:
+            text = "Cancelled"
+        elif self.__state == State.Error:
+            text = "Error state"
+        else:
+            assert False
+
+        self.info_area.setText(text)
+
+        if self.__state == State.Processing:
+            self.infostack.setCurrentIndex(1)
+        else:
+            self.infostack.setCurrentIndex(0)
+
+    def setCurrentPath(self, path):
+        """
+        Set the current root text path to path
+
+        If the path does not exists or is not a directory the current path
+        is left unchanged
+
+        Parameters
+        ----------
+        path : str
+            New root import path.
+
+        Returns
+        -------
+        status : bool
+            True if the current root import path was successfully
+            changed to path.
+        """
+        if self.currentPath is not None and path is not None and \
+                os.path.isdir(self.currentPath) and os.path.isdir(path) and \
+                os.path.samefile(self.currentPath, path):
+            return True
+
+        success = True
+        error = None
+        if path is not None:
+            if not os.path.exists(path):
+                error = "'{}' does not exist".format(path)
+                path = None
+                success = False
+            elif not os.path.isdir(path):
+                error = "'{}' is not a folder".format(path)
+                path = None
+                success = False
+
+        if error is not None:
+            self.error(error)
+            warnings.warn(error, UserWarning, stacklevel=3)
+        else:
+            self.error()
+
+        if path is not None:
+            newindex = self.addRecentPath(path)
+            self.recent_cb.setCurrentIndex(newindex)
+            if newindex >= 0:
+                self.currentPath = path
+            else:
+                self.currentPath = None
+        else:
+            self.currentPath = None
+        self.__actions.reload.setEnabled(self.currentPath is not None)
+
+        if self.__state == State.Processing:
+            self.cancel()
+
+        return success
+
+    def addRecentPath(self, path):
+        """
+        Prepend a path entry to the list of recent paths
+
+        If an entry with the same path already exists in the recent path
+        list it is moved to the first place
+
+        Parameters
+        ----------
+        path : str
+        """
+        existing = None
+        for pathitem in self.recent_paths:
+            try:
+                if os.path.samefile(pathitem.abspath, path):
+                    existing = pathitem
+                    break
+            except FileNotFoundError:
+                # file not found if the `pathitem.abspath` no longer exists
+                pass
+
+        model = self.recent_cb.model()
+
+        if existing is not None:
+            selected_index = self.recent_paths.index(existing)
+            assert model.item(selected_index).data(Qt.UserRole) is existing
+            self.recent_paths.remove(existing)
+            row = model.takeRow(selected_index)
+            self.recent_paths.insert(0, existing)
+            model.insertRow(0, row)
+        else:
+            item = RecentPath(path, None, None)
+            self.recent_paths.insert(0, item)
+            model.insertRow(0, RecentPath_asqstandarditem(item))
+        return 0
+
+    def __setRuntimeState(self, state):
+        assert state in State
+        self.setBlocking(state == State.Processing)
+        message = ""
+        if state == State.Processing:
+            assert self.__state in [State.Done,
+                                    State.NoState,
+                                    State.Error,
+                                    State.Cancelled]
+            message = "Processing"
+        elif state == State.Done:
+            assert self.__state == State.Processing
+        elif state == State.Cancelled:
+            assert self.__state == State.Processing
+            message = "Cancelled"
+        elif state == State.Error:
+            message = "Error during processing"
+        elif state == State.NoState:
+            message = ""
+        else:
+            assert False
+
+        self.__state = state
+
+        if self.__state == State.Processing:
+            self.infostack.setCurrentIndex(1)
+        else:
+            self.infostack.setCurrentIndex(0)
+
+        self.setStatusMessage(message)
+        self.__updateInfo()
+
+    def reload(self):
+        """
+        Restart the text scan task
+        """
+        if self.__state == State.Processing:
+            self.cancel()
+
+        self.corpus = None
+        self.start()
+
+    def start(self):
+        """
+        Start/execute the text indexing operation
+        """
+        self.error()
+        self.Warning.clear()
+        self.progress_widget.setValue(0)
+
+        self.__invalidated = False
+        if self.currentPath is None:
+            return
+
+        if self.__state == State.Processing:
+            assert self.__pendingTask is not None
+            log.info("Starting a new task while one is in progress. "
+                     "Cancel the existing task (dir:'{}')"
+                     .format(self.__pendingTask.startdir))
+            self.cancel()
+
+        startdir = self.currentPath
+
+        self.__setRuntimeState(State.Processing)
+
+        report_progress = methodinvoke(
+            self, "__onReportProgress", (object,))
+
+        task = ImportDocuments(startdir, report_progress=report_progress)
+
+        # collect the task state in one convenient place
+        self.__pendingTask = taskstate = namespace(
+            task=task,
+            startdir=startdir,
+            future=None,
+            watcher=None,
+            cancelled=False,
+            cancel=None,
+        )
+
+        def cancel():
+            # Cancel the task and disconnect
+            if taskstate.future.cancel():
+                pass
+            else:
+                taskstate.task.cancelled = True
+                taskstate.cancelled = True
+                try:
+                    taskstate.future.result(timeout=0)
+                except UserInterruptError:
+                    pass
+                except TimeoutError:
+                    log.info("The task did not stop in in a timely manner")
+            taskstate.watcher.finished.disconnect(self.__onRunFinished)
+
+        taskstate.cancel = cancel
+
+        def run_text_scan_task_interupt():
+            try:
+                return task.run()
+            except UserInterruptError:
+                # Suppress interrupt errors, so they are not logged
+                return
+
+        taskstate.future = self.__executor.submit(run_text_scan_task_interupt)
+        taskstate.watcher = FutureWatcher(taskstate.future)
+        taskstate.watcher.finished.connect(self.__onRunFinished)
+
+    @Slot()
+    def __onRunFinished(self):
+        assert QThread.currentThread() is self.thread()
+        assert self.__state == State.Processing
+        assert self.__pendingTask is not None
+        assert self.sender() is self.__pendingTask.watcher
+        assert self.__pendingTask.future.done()
+        task = self.__pendingTask
+        self.__pendingTask = None
+
+        try:
+            corpus, errors = task.future.result()
+        except Exception:
+            sys.excepthook(*sys.exc_info())
+            state = State.Error
+            corpus = None
+            errors = []
+            self.error(traceback.format_exc())
+        else:
+            state = State.Done
+            self.error()
+
+        if corpus:
+            self.n_text_data = len(corpus)
+            self.n_text_categories = len(corpus.domain.class_var.values)\
+                if corpus.domain.class_var else 0
+
+        self.corpus = corpus
+        self.n_skipped = len(errors)
+
+        if len(errors):
+            self.Warning.read_error("Some files" if len(errors) > 1 else "One file")
+
+        self.__setRuntimeState(state)
+        self.commit()
+
+    def cancel(self):
+        """
+        Cancel current pending task (if any).
+        """
+        if self.__state == State.Processing:
+            assert self.__pendingTask is not None
+            self.__pendingTask.cancel()
+            self.__pendingTask = None
+            self.__setRuntimeState(State.Cancelled)
+
+    @Slot(object)
+    def __onReportProgress(self, arg):
+        # report on scan progress from a worker thread
+        # arg must be a namespace(count: int, lastpath: str)
+        assert QThread.currentThread() is self.thread()
+        if self.__state == State.Processing:
+            self.pathlabel.setText(prettifypath(arg.lastpath))
+            self.progress_widget.setValue(arg.progress)
+            self.progress_widget.setValue(100 * arg.progress)
+
+    def commit(self):
+        """
+        Create and commit a Corpus from the collected text meta data.
+        """
+        self.Outputs.data.send(self.corpus)
+
+    def onDeleteWidget(self):
+        self.cancel()
+        self.__executor.shutdown(wait=True)
+        self.__invalidated = False
+
+    def eventFilter(self, receiver, event):
+        # re-implemented from QWidget
+        # intercept and process drag drop events on the recent directory
+        # selection combo box
+        def dirpath(event):
+            # type: (QDropEvent) -> Optional[str]
+            """Return the directory from a QDropEvent."""
+            data = event.mimeData()
+            urls = data.urls()
+            if len(urls) == 1:
+                url = urls[0]
+                path = url.toLocalFile()
+                if os.path.isdir(path):
+                    return path
+            return None
+
+        if receiver is self.recent_cb and \
+                event.type() in {QEvent.DragEnter, QEvent.DragMove,
+                                 QEvent.Drop}:
+            assert isinstance(event, QDropEvent)
+            path = dirpath(event)
+            if path is not None and event.possibleActions() & Qt.LinkAction:
+                event.setDropAction(Qt.LinkAction)
+                event.accept()
+                if event.type() == QEvent.Drop:
+                    self.setCurrentPath(path)
+                    self.start()
+            else:
+                event.ignore()
+            return True
+
+        return super().eventFilter(receiver, event)
+
+
+class UserInterruptError(BaseException):
+    """
+    A BaseException subclass used for cooperative task/thread cancellation
+    """
+    pass
+
+
+def main(argv=sys.argv):
+    app = QApplication(list(argv))
+    argv = app.arguments()
+    if len(argv) > 1:
+        path = argv[1]
+    else:
+        path = None
+    w = OWImportDocuments()
+    w.show()
+    w.raise_()
+
+    if path is not None:
+        w.setCurrentPath(path)
+
+    app.exec_()
+    w.saveSettings()
+    w.onDeleteWidget()
+    return 0
+
+if __name__ == "__main__":
+    sys.exit(main())


Test patch: