Repository: ad-freiburg/qlever (language: python)

Issue:
Predicates of the form `@de@<http://rdaregistry.info/Elements/u/P60470>` in query result
When indexing data with QLever, sometimes strange values are returned for the predicates. To reproduce:

```shell
curl https://qlever.cs.uni-freiburg.de/api/dnb --data 'query=SELECT%20%3Fp%20%3Fo%20WHERE%0A%20%7B%0A%20%20%3Chttps%3A//d-nb.info/454818084%3E%20%3Fp%20%3Fo%20.%0A%20%7D' -X POST
```

This produces values that look like:
```json
{
    "head": {
        "vars": [
            "p",
            "o"
        ]
    },
    "results": {
        "bindings": [
            {
                "o": {
                    "type": "bnode",
                    "value": "bn27771275"
                },
                "p": {
                    "type": "uri",
                    "value": "http://id.loc.gov/vocabulary/relators/aut"
                }
            },
---8<----   etc.  ---8<---- 
            {
                "o": {
                    "type": "literal",
                    "value": "mit Abb.",
                    "xml:lang": "de"
                },
                "p": {
                    "type": "literal",
                    "value": "@de@<http://rdaregistry.info/Elements/u/P60470>"
                }
            }
        ]
    }
}
```

Notice the weird literal shown as a predicate, as if there was a linebreak issue in parsing the source data. Here we are showing one of the example datasets provided by the authors, the DNB set.

But this has also happened in some cases for triples that I have created myself, where I could verify that the input triples did not seem malformed.





Discussion / hints:
Just to make sure that I understand your question correctly:
Are you wondering about the strange predicate `@de@<http:....>`  which is (wrongly) marked as a literal?
Those predicates and their triples are artifacts of QLever's implementation of language filters. They are currently leaked for queries without a fixed predicate in all triples (your predicate is a variable `?p`).
It is on our list to fully hide these triples (wrt to exports/query results/ statistics) such that this can't happen.

Thanks for reporting this, we now have an issue to track. It is also interesting to see, how these triples currently interact with the JSON exporter.
Please confirm that I have correctly identified your issue.
Thanks for the answer, yes I was wondering about the predicates being marked as literals in the SPARQL query results.
First time I saw something like this was in a CMS vendor publishing their Linked Data one year ago. Filed a bug report there, but  never heard back from them. Now we know that they are using QLever as a triplestore "under the covers" ;-) 

I am new to QLever, just ran some tests ingesting data last week and noticed the strange output. At first I thought it was something wrong in my own data, but then noticed it was also in the sample datasets provided. (and I recalled seeing this previously elsewhere, as mentioned)

Look forward to learning more about the internals later, good to know that this is an implementation artefact and will eventually disappear.
@epoz Interesting story, which CMS vendor was this? Is there a issue on their GitHub or was this some internal bug tracker?
It is a Dutch CMS for cultural heritage content: https://kleksi.com/nl/home 
Their service is not open source, and does not have a bug tracker. I submitted a request to their support@ mail, and received a reply via email later from the owner that they would pass my query on to their tech department. Never heard back from them, and I did not consider it important enough to follow up.

[This was the event](https://netwerkdigitaalerfgoed.nl/nieuws/hackalod-2023-dit-zijn-de-toepassingen-met-erfgoeddata-voor-publiek-en-collegas/) where we were spelunking for interesting data sources.

Fix patch:
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 2f67ab33a..7d4e407b9 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -421,8 +421,5 @@ target_precompile_headers(ServerMain REUSE_FROM engine)
 add_executable(VocabularyMergerMain src/VocabularyMergerMain.cpp)
 qlever_target_link_libraries(VocabularyMergerMain index ${CMAKE_THREAD_LIBS_INIT})
 
-add_executable(PermutationExporterMain src/index/PermutationExporterMain.cpp)
-qlever_target_link_libraries(PermutationExporterMain index ${CMAKE_THREAD_LIBS_INIT})
-
 add_executable(PrintIndexVersionMain src/PrintIndexVersionMain.cpp)
 qlever_target_link_libraries(PrintIndexVersionMain util)
diff --git a/src/engine/CMakeLists.txt b/src/engine/CMakeLists.txt
index 426316965..878bc60b3 100644
--- a/src/engine/CMakeLists.txt
+++ b/src/engine/CMakeLists.txt
@@ -12,5 +12,5 @@ add_library(engine
         Values.cpp Bind.cpp Minus.cpp RuntimeInformation.cpp CheckUsePatternTrick.cpp
         VariableToColumnMap.cpp ExportQueryExecutionTrees.cpp
         CartesianProductJoin.cpp TextIndexScanForWord.cpp TextIndexScanForEntity.cpp
-        TextLimit.cpp LocalVocabEntry.cpp LazyGroupBy.cpp GroupByHashMapOptimization.cpp SpatialJoin.cpp)
+        TextLimit.cpp LazyGroupBy.cpp GroupByHashMapOptimization.cpp SpatialJoin.cpp)
 qlever_target_link_libraries(engine util index parser sparqlExpressions http SortPerformanceEstimator Boost::iostreams)
diff --git a/src/engine/GroupBy.cpp b/src/engine/GroupBy.cpp
index c7f79f1e0..2088a1220 100644
--- a/src/engine/GroupBy.cpp
+++ b/src/engine/GroupBy.cpp
@@ -708,8 +708,6 @@ std::optional<IdTable> GroupBy::computeGroupByForFullIndexScan() const {
 
   _subtree->getRootOperation()->updateRuntimeInformationWhenOptimizedOut({});
 
-  auto ignoredRanges =
-      getIndex().getImpl().getIgnoredIdRanges(permutationEnum.value()).first;
   const auto& permutation =
       getExecutionContext()->getIndex().getPimpl().getPermutation(
           permutationEnum.value());
@@ -717,13 +715,6 @@ std::optional<IdTable> GroupBy::computeGroupByForFullIndexScan() const {
   if (numCounts == 0) {
     table.setColumnSubset({{0}});
   }
-  // TODO<joka921> This is only semi-efficient.
-  auto end = std::ranges::remove_if(table, [&ignoredRanges](const auto& row) {
-    return std::ranges::any_of(ignoredRanges, [id = row[0]](const auto& pair) {
-      return id >= pair.first && id < pair.second;
-    });
-  });
-  table.resize(end.begin() - table.begin());
 
   // TODO<joka921> This optimization should probably also apply if
   // the query is `SELECT DISTINCT ?s WHERE {?s ?p ?o} ` without a
diff --git a/src/engine/IndexScan.cpp b/src/engine/IndexScan.cpp
index 46ab11077..a9947e938 100644
--- a/src/engine/IndexScan.cpp
+++ b/src/engine/IndexScan.cpp
@@ -11,7 +11,6 @@
 #include <string>
 
 #include "index/IndexImpl.h"
-#include "index/TriplesView.h"
 #include "parser/ParsedQuery.h"
 
 using std::string;
@@ -161,13 +160,8 @@ ProtoResult IndexScan::computeResult(bool requestLaziness) {
   using enum Permutation::Enum;
   idTable.setNumColumns(numVariables_);
   const auto& index = _executionContext->getIndex();
-  if (numVariables_ < 3 || !additionalColumns().empty()) {
-    idTable = index.scan(getScanSpecification(), permutation_,
-                         additionalColumns(), cancellationHandle_, getLimit());
-  } else {
-    AD_CORRECTNESS_CHECK(numVariables_ == 3);
-    computeFullScan(&idTable, permutation_);
-  }
+  idTable = index.scan(getScanSpecification(), permutation_,
+                       additionalColumns(), cancellationHandle_, getLimit());
   AD_CORRECTNESS_CHECK(idTable.numColumns() == getResultWidth());
   LOG(DEBUG) << "IndexScan result computation done.\n";
   checkCancellation();
@@ -178,19 +172,7 @@ ProtoResult IndexScan::computeResult(bool requestLaziness) {
 // _____________________________________________________________________________
 size_t IndexScan::computeSizeEstimate() const {
   AD_CORRECTNESS_CHECK(_executionContext);
-  // We have to do a simple scan anyway so might as well do it now
-  if (numVariables_ < 3) {
-    return getIndex().getResultSizeOfScan(getScanSpecification(), permutation_);
-  } else {
-    // The triple consists of three variables.
-    // TODO<joka921> As soon as all implementations of a full index scan
-    // (Including the "dummy joins" in Join.cpp) consistently exclude the
-    // internal triples, this estimate should be changed to only return
-    // the number of triples in the actual knowledge graph (excluding the
-    // internal triples).
-    AD_CORRECTNESS_CHECK(numVariables_ == 3);
-    return getIndex().numTriples().normalAndInternal_();
-  }
+  return getIndex().getResultSizeOfScan(getScanSpecification(), permutation_);
 }
 
 // _____________________________________________________________________________
@@ -223,44 +205,6 @@ void IndexScan::determineMultiplicities() {
   AD_CONTRACT_CHECK(multiplicity_.size() == getResultWidth());
 }
 
-// ________________________________________________________________________
-void IndexScan::computeFullScan(IdTable* result,
-                                const Permutation::Enum permutation) const {
-  auto [ignoredRanges, isTripleIgnored] =
-      getIndex().getImpl().getIgnoredIdRanges(permutation);
-
-  result->setNumColumns(3);
-
-  // This implementation computes the complete knowledge graph, except the
-  // internal triples.
-  uint64_t resultSize = getIndex().numTriples().normal;
-  if (getLimit()._limit.has_value() && getLimit()._limit < resultSize) {
-    resultSize = getLimit()._limit.value();
-  }
-
-  // TODO<joka921> Implement OFFSET
-  if (getLimit()._offset != 0) {
-    throw NotSupportedException{
-        "Scanning the complete index with an OFFSET clause is currently not "
-        "supported by QLever"};
-  }
-  result->reserve(resultSize);
-  auto table = std::move(*result).toStatic<3>();
-  size_t i = 0;
-  const auto& permutationImpl =
-      getExecutionContext()->getIndex().getImpl().getPermutation(permutation);
-  auto triplesView = TriplesView(permutationImpl, cancellationHandle_,
-                                 ignoredRanges, isTripleIgnored);
-  for (const auto& triple : triplesView) {
-    if (i >= resultSize) {
-      break;
-    }
-    table.push_back(triple);
-    ++i;
-  }
-  *result = std::move(table).toDynamic();
-}
-
 // ___________________________________________________________________________
 std::array<const TripleComponent* const, 3> IndexScan::getPermutedTriple()
     const {
diff --git a/src/engine/IndexScan.h b/src/engine/IndexScan.h
index d63f633f1..bf73b2262 100644
--- a/src/engine/IndexScan.h
+++ b/src/engine/IndexScan.h
@@ -124,8 +124,6 @@ class IndexScan final : public Operation {
 
   vector<QueryExecutionTree*> getChildren() override { return {}; }
 
-  void computeFullScan(IdTable* result, Permutation::Enum permutation) const;
-
   size_t computeSizeEstimate() const;
 
   std::string getCacheKeyImpl() const override;
diff --git a/src/global/IndexTypes.h b/src/global/IndexTypes.h
index 372e9e57c..08ee960d0 100644
--- a/src/global/IndexTypes.h
+++ b/src/global/IndexTypes.h
@@ -4,9 +4,9 @@
 
 #pragma once
 
-#include "engine/LocalVocabEntry.h"
 #include "global/TypedIndex.h"
 #include "global/VocabIndex.h"
+#include "index/LocalVocabEntry.h"
 
 // Typedefs for several kinds of typed indices that are used across QLever.
 
diff --git a/src/index/CMakeLists.txt b/src/index/CMakeLists.txt
index aaed46ca5..1fc877372 100644
--- a/src/index/CMakeLists.txt
+++ b/src/index/CMakeLists.txt
@@ -6,5 +6,5 @@ add_library(index
         DocsDB.cpp FTSAlgorithms.cpp
         PrefixHeuristic.cpp CompressedRelation.cpp
         PatternCreator.cpp ScanSpecification.cpp
-        DeltaTriples.cpp)
+        DeltaTriples.cpp LocalVocabEntry.cpp)
 qlever_target_link_libraries(index util parser vocabulary ${STXXL_LIBRARIES})
diff --git a/src/index/ConstantsIndexBuilding.h b/src/index/ConstantsIndexBuilding.h
index 4653163a3..5f7643484 100644
--- a/src/index/ConstantsIndexBuilding.h
+++ b/src/index/ConstantsIndexBuilding.h
@@ -62,6 +62,9 @@ constexpr inline std::string_view PARTIAL_MMAP_IDS = ".tmp.partial-ids-mmap.";
 constexpr inline std::string_view TMP_BASENAME_COMPRESSION =
     ".tmp.for-prefix-compression";
 
+// _________________________________________________________________
+constexpr inline std::string_view INTERNAL_INDEX_INFIX = ".internal";
+
 // _________________________________________________________________
 // The degree of parallelism that is used for the index building step, where the
 // unique elements of the vocabulary are identified via hash maps. Typically, 6
diff --git a/src/index/IndexBuilderTypes.h b/src/index/IndexBuilderTypes.h
index e0c14cd17..173e32309 100644
--- a/src/index/IndexBuilderTypes.h
+++ b/src/index/IndexBuilderTypes.h
@@ -262,14 +262,7 @@ auto getIdMapLambdas(
    * - All Ids are assigned according to itemArray[idx]
    */
   const auto itemMapLamdaCreator = [&itemArray, indexPtr](const size_t idx) {
-    auto& map = *itemArray[idx];
-    // Resolve the special IDs of the default and internal graph to their actual
-    // IDs. This is precomputed for efficiency gains.
-    auto internalGraphId =
-        map.getId(qlever::specialIds().at(INTERNAL_GRAPH_IRI));
-    auto defaultGraphId = map.getId(qlever::specialIds().at(DEFAULT_GRAPH_IRI));
-    return [&map = *itemArray[idx], indexPtr, internalGraphId,
-            defaultGraphId](ad_utility::Rvalue auto&& tr) {
+    return [&map = *itemArray[idx], indexPtr](ad_utility::Rvalue auto&& tr) {
       auto lt = indexPtr->tripleToInternalRepresentation(AD_FWD(tr));
       OptionalIds res;
       // get Ids for the actual triple and store them in the result.
@@ -293,24 +286,19 @@ auto getIdMapLambdas(
                       " The following lines probably have to be changed when "
                       "the number of payload columns changes");
         // extra triple <subject> @language@<predicate> <object>
-        // The additional triples have the graph ID of the internal graph if the
-        // triple was in the default/fallback graph, else they keep their graph
-        // ID.
-        // TODO<joka921> Maybe we should have an `internalGraph` per graph, but
-        // this requires further work. The current approach at least keeps the
-        // language filters working in combination with named graphs and doesn't
-        // add further inconsistencies.
+        // The additional triples have the same graph ID as the original triple.
+        // This makes optimizations such as language filters also work with
+        // named graphs. Note that we have a different mechanism in place to
+        // distinguish between normal and internal triples.
         auto tripleGraphId = res[0].value()[ADDITIONAL_COLUMN_GRAPH_ID];
-        auto addedTripleGraphId =
-            tripleGraphId == defaultGraphId ? internalGraphId : tripleGraphId;
         res[1].emplace(
-            Arr{spoIds[0], langTaggedPredId, spoIds[2], addedTripleGraphId});
+            Arr{spoIds[0], langTaggedPredId, spoIds[2], tripleGraphId});
         // extra triple <object> ql:language-tag <@language>
         res[2].emplace(Arr{spoIds[2],
                            map.getId(TripleComponent{
                                ad_utility::triple_component::Iri::fromIriref(
                                    LANGUAGE_PREDICATE)}),
-                           langTagId, addedTripleGraphId});
+                           langTagId, tripleGraphId});
       }
       return res;
     };
diff --git a/src/index/IndexFormatVersion.h b/src/index/IndexFormatVersion.h
index 2a59bedb3..84bee0831 100644
--- a/src/index/IndexFormatVersion.h
+++ b/src/index/IndexFormatVersion.h
@@ -36,5 +36,5 @@ struct IndexFormatVersion {
 // The actual index version. Change it once the binary format of the index
 // changes.
 inline const IndexFormatVersion& indexFormatVersion{
-    1506, DateYearOrDuration{Date{2024, 9, 27}}};
+    1532, DateYearOrDuration{Date{2024, 10, 4}}};
 }  // namespace qlever
diff --git a/src/index/IndexImpl.cpp b/src/index/IndexImpl.cpp
index 3000d11e8..35e6f773c 100644
--- a/src/index/IndexImpl.cpp
+++ b/src/index/IndexImpl.cpp
@@ -19,8 +19,6 @@
 #include "engine/AddCombinedRowToTable.h"
 #include "engine/CallFixedSize.h"
 #include "index/IndexFormatVersion.h"
-#include "index/PrefixHeuristic.h"
-#include "index/TriplesView.h"
 #include "index/VocabularyMerger.h"
 #include "parser/ParallelParseBuffer.h"
 #include "util/BatchedPipeline.h"
@@ -54,9 +52,16 @@ IndexBuilderDataAsFirstPermutationSorter IndexImpl::createIdTriplesAndVocab(
   auto indexBuilderData =
       passFileForVocabulary(std::move(parser), numTriplesPerBatch_);
 
+  auto isQleverInternalTriple = [&indexBuilderData](const auto& triple) {
+    auto internal = [&indexBuilderData](Id id) {
+      return indexBuilderData.vocabularyMetaData_.isQleverInternalId(id);
+    };
+    return internal(triple[0]) || internal(triple[1]) || internal(triple[2]);
+  };
+
   auto firstSorter = convertPartialToGlobalIds(
       *indexBuilderData.idTriples, indexBuilderData.actualPartialSizes,
-      NUM_TRIPLES_PER_PARTIAL_VOCAB);
+      NUM_TRIPLES_PER_PARTIAL_VOCAB, isQleverInternalTriple);
 
   return {indexBuilderData, std::move(firstSorter)};
 }
@@ -166,7 +171,7 @@ auto fixBlockAfterPatternJoin(auto block) {
 std::unique_ptr<ExternalSorter<SortByPSO, NumColumnsIndexBuilding + 2>>
 IndexImpl::buildOspWithPatterns(
     PatternCreator::TripleSorter sortersFromPatternCreator,
-    auto isQleverInternalTriple) {
+    auto& internalTripleSorter) {
   auto&& [hasPatternPredicateSortedByPSO, secondSorter] =
       sortersFromPatternCreator;
   // We need the patterns twice: once for the additional column, and once for
@@ -245,8 +250,7 @@ IndexImpl::buildOspWithPatterns(
   auto thirdSorter =
       makeSorterPtr<ThirdPermutation, NumColumnsIndexBuilding + 2>("third");
   createSecondPermutationPair(NumColumnsIndexBuilding + 2,
-                              isQleverInternalTriple, std::move(blockGenerator),
-                              *thirdSorter);
+                              std::move(blockGenerator), *thirdSorter);
   secondSorter->clear();
   // Add the `ql:has-pattern` predicate to the sorter such that it will become
   // part of the PSO and POS permutation.
@@ -254,20 +258,44 @@ IndexImpl::buildOspWithPatterns(
             << " triples to the POS and PSO permutation for "
                "the internal `ql:has-pattern` ..."
             << std::endl;
-  auto noPattern = Id::makeFromInt(NO_PATTERN);
   static_assert(NumColumnsIndexBuilding == 4,
                 "When adding additional payload columns, the following code "
                 "has to be changed");
   Id internalGraph = idOfInternalGraphDuringIndexBuilding_.value();
+  // Note: We are getting the patterns sorted by PSO and then sorting them again
+  // by PSO.
+  // TODO<joka921> Simply get the output unsorted (should be cheaper).
   for (const auto& row : hasPatternPredicateSortedByPSO->sortedView()) {
-    // The repetition of the pattern index (`row[2]`) for the fifth column is
-    // useful for generic unit testing, but not needed otherwise.
-    thirdSorter->push(
-        std::array{row[0], row[1], row[2], internalGraph, row[2], noPattern});
+    internalTripleSorter.push(
+        std::array{row[0], row[1], row[2], internalGraph});
   }
   hasPatternPredicateSortedByPSO->clear();
   return thirdSorter;
 }
+
+// _____________________________________________________________________________
+std::pair<size_t, size_t> IndexImpl::createInternalPSOandPOS(
+    auto&& internalTriplesPsoSorter) {
+  auto onDiskBaseBackup = onDiskBase_;
+  auto configurationJsonBackup = configurationJson_;
+  onDiskBase_.append(INTERNAL_INDEX_INFIX);
+  auto internalTriplesUnique = ad_utility::uniqueBlockView(
+      internalTriplesPsoSorter.template getSortedBlocks<0>());
+  createPSOAndPOSImpl(NumColumnsIndexBuilding, std::move(internalTriplesUnique),
+                      false);
+  onDiskBase_ = std::move(onDiskBaseBackup);
+  // The "normal" triples from the "internal" index builder are actually
+  // internal.
+  size_t numTriplesInternal =
+      static_cast<NumNormalAndInternal>(configurationJson_["num-triples"])
+          .normal;
+  size_t numPredicatesInternal =
+      static_cast<NumNormalAndInternal>(configurationJson_["num-predicates"])
+          .normal;
+  configurationJson_ = std::move(configurationJsonBackup);
+  return {numTriplesInternal, numPredicatesInternal};
+}
+
 // _____________________________________________________________________________
 void IndexImpl::createFromFile(const string& filename, Index::Filetype type) {
   if (!loadAllPermutations_ && usePatterns_) {
@@ -286,42 +314,46 @@ void IndexImpl::createFromFile(const string& filename, Index::Filetype type) {
   // case any of the permutations fail.
   writeConfiguration();
 
-  auto isQleverInternalId = [&indexBuilderData](const auto& id) {
-    // The special internal IDs like `ql:has-pattern` (see `SpecialIds.h`)
-    // have the datatype `UNDEFINED`.
-    return indexBuilderData.vocabularyMetaData_.isQleverInternalId(id) ||
-           id.getDatatype() == Datatype::Undefined;
-  };
+  auto& firstSorter = *indexBuilderData.sorter_.firstPermutationSorter_;
+
+  size_t numTriplesInternal = 0;
+  size_t numPredicatesInternal = 0;
 
-  auto isQleverInternalTriple = [isQleverInternalId](const auto& triple) {
-    const auto& i = isQleverInternalId;
-    return i(triple[0]) || i(triple[1]) || i(triple[2]);
+  // Create the internal PSO and POS permutations. This has to be called AFTER
+  // all triples have been added to the `internalTriplesPso_` sorter, in
+  // particular, after the patterns have been created.
+  auto createInternalPsoAndPosAndSetMetadata = [this, &numTriplesInternal,
+                                                &numPredicatesInternal,
+                                                &indexBuilderData]() {
+    std::tie(numTriplesInternal, numPredicatesInternal) =
+        createInternalPSOandPOS(*indexBuilderData.sorter_.internalTriplesPso_);
   };
 
-  auto& firstSorter = *indexBuilderData.sorter_;
   // For the first permutation, perform a unique.
   auto firstSorterWithUnique =
       ad_utility::uniqueBlockView(firstSorter.getSortedOutput());
 
   if (!loadAllPermutations_) {
+    createInternalPsoAndPosAndSetMetadata();
     // Only two permutations, no patterns, in this case the `firstSorter` is a
     // PSO sorter, and `createPermutationPair` creates PSO/POS permutations.
-    createFirstPermutationPair(NumColumnsIndexBuilding, isQleverInternalTriple,
+    createFirstPermutationPair(NumColumnsIndexBuilding,
                                std::move(firstSorterWithUnique));
     configurationJson_["has-all-permutations"] = false;
-  } else if (loadAllPermutations_ && !usePatterns_) {
-    // Without patterns we explicitly have to pass in the next sorters to all
+  } else if (!usePatterns_) {
+    createInternalPsoAndPosAndSetMetadata();
+    // Without patterns, we explicitly have to pass in the next sorters to all
     // permutation creating functions.
     auto secondSorter = makeSorter<SecondPermutation>("second");
-    createFirstPermutationPair(NumColumnsIndexBuilding, isQleverInternalTriple,
+    createFirstPermutationPair(NumColumnsIndexBuilding,
                                std::move(firstSorterWithUnique), secondSorter);
     firstSorter.clearUnderlying();
 
     auto thirdSorter = makeSorter<ThirdPermutation>("third");
-    createSecondPermutationPair(NumColumnsIndexBuilding, isQleverInternalTriple,
+    createSecondPermutationPair(NumColumnsIndexBuilding,
                                 secondSorter.getSortedBlocks<0>(), thirdSorter);
     secondSorter.clear();
-    createThirdPermutationPair(NumColumnsIndexBuilding, isQleverInternalTriple,
+    createThirdPermutationPair(NumColumnsIndexBuilding,
                                thirdSorter.getSortedBlocks<0>());
     configurationJson_["has-all-permutations"] = true;
   } else {
@@ -329,23 +361,39 @@ void IndexImpl::createFromFile(const string& filename, Index::Filetype type) {
     // `createFirstPermutationPair` function returns the next sorter, already
     // enriched with the patterns of the subjects in the triple.
     auto patternOutput = createFirstPermutationPair(
-        NumColumnsIndexBuilding, isQleverInternalTriple,
-        std::move(firstSorterWithUnique));
+        NumColumnsIndexBuilding, std::move(firstSorterWithUnique));
     firstSorter.clearUnderlying();
-    auto thirdSorterPtr = buildOspWithPatterns(std::move(patternOutput.value()),
-                                               isQleverInternalTriple);
+    auto thirdSorterPtr =
+        buildOspWithPatterns(std::move(patternOutput.value()),
+                             *indexBuilderData.sorter_.internalTriplesPso_);
+    createInternalPsoAndPosAndSetMetadata();
     createThirdPermutationPair(NumColumnsIndexBuilding + 2,
-                               isQleverInternalTriple,
+
                                thirdSorterPtr->template getSortedBlocks<0>());
     configurationJson_["has-all-permutations"] = true;
   }
 
-  // Dump the configuration again in case the permutations have added some
-  // information.
-  writeConfiguration();
+  addInternalStatisticsToConfiguration(numTriplesInternal,
+                                       numPredicatesInternal);
   LOG(INFO) << "Index build completed" << std::endl;
 }
 
+// _____________________________________________________________________________
+void IndexImpl::addInternalStatisticsToConfiguration(
+    size_t numTriplesInternal, size_t numPredicatesInternal) {
+  // Combine the number of normal triples and predicates with their internal
+  // counterparts.
+  auto numTriples =
+      static_cast<NumNormalAndInternal>(configurationJson_["num-triples"]);
+  auto numPredicates =
+      static_cast<NumNormalAndInternal>(configurationJson_["num-predicates"]);
+  numTriples.internal = numTriplesInternal;
+  numPredicates.internal = numPredicatesInternal;
+  configurationJson_["num-triples"] = numTriples;
+  configurationJson_["num-predicates"] = numPredicates;
+  writeConfiguration();
+}
+
 // _____________________________________________________________________________
 IndexBuilderDataAsStxxlVector IndexImpl::passFileForVocabulary(
     std::shared_ptr<RdfParserBase> parser, size_t linesPerPartial) {
@@ -500,10 +548,10 @@ IndexBuilderDataAsStxxlVector IndexImpl::passFileForVocabulary(
 }
 
 // _____________________________________________________________________________
-std::unique_ptr<ad_utility::CompressedExternalIdTableSorterTypeErased>
-IndexImpl::convertPartialToGlobalIds(
+auto IndexImpl::convertPartialToGlobalIds(
     TripleVec& data, const vector<size_t>& actualLinesPerPartial,
-    size_t linesPerPartial) {
+    size_t linesPerPartial, auto isQLeverInternalTriple)
+    -> FirstPermutationSorterAndInternalTriplesAsPso {
   LOG(INFO) << "Converting triples from local IDs to global IDs ..."
             << std::endl;
   LOG(DEBUG) << "Triples per partial vocabulary: " << linesPerPartial
@@ -519,10 +567,17 @@ IndexImpl::convertPartialToGlobalIds(
       return makeSorterPtr<SortByPSO>("first");
     }
   }();
+  auto internalTriplesPtr =
+      makeSorterPtr<SortByPSO, NumColumnsIndexBuilding>("internalTriples");
   auto& result = *resultPtr;
+  auto& internalResult = *internalTriplesPtr;
   auto triplesGenerator = data.getRows();
   auto it = triplesGenerator.begin();
   using Buffer = IdTableStatic<NumColumnsIndexBuilding>;
+  struct Buffers {
+    IdTableStatic<NumColumnsIndexBuilding> triples_;
+    IdTableStatic<NumColumnsIndexBuilding> internalTriples_;
+  };
   using Map = ad_utility::HashMap<Id, Id>;
 
   ad_utility::TaskQueue<true> lookupQueue(30, 10,
@@ -555,13 +610,18 @@ IndexImpl::convertPartialToGlobalIds(
   size_t numTriplesConverted = 0;
   ad_utility::ProgressBar progressBar{numTriplesConverted,
                                       "Triples converted: "};
-  auto getWriteTask = [&result, &numTriplesConverted,
-                       &progressBar](Buffer triples) {
-    return [&result, &numTriplesConverted, &progressBar,
+  auto getWriteTask = [&result, &internalResult, &numTriplesConverted,
+                       &progressBar](Buffers buffers) {
+    return [&result, &internalResult, &numTriplesConverted, &progressBar,
             triples = std::make_shared<IdTableStatic<0>>(
-                std::move(triples).toDynamic())] {
+                std::move(buffers.triples_).toDynamic()),
+            internalTriples = std::make_shared<IdTableStatic<0>>(
+                std::move(buffers.internalTriples_).toDynamic())] {
       result.pushBlock(*triples);
+      internalResult.pushBlock(*internalTriples);
+
       numTriplesConverted += triples->size();
+      numTriplesConverted += internalTriples->size();
       if (progressBar.update()) {
         LOG(INFO) << progressBar.getProgressString() << std::flush;
       }
@@ -571,15 +631,32 @@ IndexImpl::convertPartialToGlobalIds(
   // Return a lambda that for each of the `triples` transforms its partial to
   // global IDs using the `idMap`. The map is passed as a `shared_ptr` because
   // multiple batches need access to the same map.
-  auto getLookupTask = [&writeQueue, &transformTriple, &getWriteTask](
-                           Buffer triples, std::shared_ptr<Map> idMap) {
+  auto getLookupTask = [&isQLeverInternalTriple, &writeQueue, &transformTriple,
+                        &getWriteTask](Buffer triples,
+                                       std::shared_ptr<Map> idMap) {
     return
-        [&writeQueue, triples = std::make_shared<Buffer>(std::move(triples)),
+        [&isQLeverInternalTriple, &writeQueue,
+         triples = std::make_shared<Buffer>(std::move(triples)),
          idMap = std::move(idMap), &getWriteTask, &transformTriple]() mutable {
           for (Buffer::row_reference triple : *triples) {
             transformTriple(triple, *idMap);
           }
-          writeQueue.push(getWriteTask(std::move(*triples)));
+          auto [beginInternal, endInternal] = std::ranges::partition(
+              *triples, [&isQLeverInternalTriple](const auto& row) {
+                return !isQLeverInternalTriple(row);
+              });
+          IdTableStatic<NumColumnsIndexBuilding> internalTriples(
+              triples->getAllocator());
+          // TODO<joka921> We could leave the partitioned complete block as is,
+          // and change the interface of the compressed sorters s.t. we can
+          // push only a part of a block. We then would safe the copy of the
+          // internal triples here, but I am not sure whether this is worth it.
+          internalTriples.insertAtEnd(beginInternal, endInternal);
+          triples->resize(beginInternal - triples->begin());
+
+          Buffers buffers{std::move(*triples), std::move(internalTriples)};
+
+          writeQueue.push(getWriteTask(std::move(buffers)));
         };
   };
 
@@ -638,7 +715,7 @@ IndexImpl::convertPartialToGlobalIds(
   lookupQueue.finish();
   writeQueue.finish();
   LOG(INFO) << progressBar.getFinalProgressString() << std::flush;
-  return resultPtr;
+  return {std::move(resultPtr), std::move(internalTriplesPtr)};
 }
 
 // _____________________________________________________________________________
@@ -751,14 +828,27 @@ void IndexImpl::createFromOnDiskIndex(const string& onDiskBase) {
   LOG(DEBUG) << "Number of words in internal and external vocabulary: "
              << vocab_.size() << std::endl;
 
-  pso_.loadFromDisk(onDiskBase_);
-  pos_.loadFromDisk(onDiskBase_);
+  auto range1 =
+      vocab_.prefixRanges(absl::StrCat("<", INTERNAL_PREDICATE_PREFIX));
+  auto range2 = vocab_.prefixRanges("@");
+  auto isInternalId = [range1, range2](Id id) {
+    // TODO<joka921> What about internal vocab stuff for update queries? this
+    // has to be added also to the external permutation.
+    if (id.getDatatype() != Datatype::VocabIndex) {
+      return false;
+    }
+    return range1.contain(id.getVocabIndex()) ||
+           range2.contain(id.getVocabIndex());
+  };
+
+  pso_.loadFromDisk(onDiskBase_, isInternalId, true);
+  pos_.loadFromDisk(onDiskBase_, isInternalId, true);
 
   if (loadAllPermutations_) {
-    ops_.loadFromDisk(onDiskBase_);
-    osp_.loadFromDisk(onDiskBase_);
-    spo_.loadFromDisk(onDiskBase_);
-    sop_.loadFromDisk(onDiskBase_);
+    ops_.loadFromDisk(onDiskBase_, isInternalId);
+    osp_.loadFromDisk(onDiskBase_, isInternalId);
+    spo_.loadFromDisk(onDiskBase_, isInternalId);
+    sop_.loadFromDisk(onDiskBase_, isInternalId);
   } else {
     LOG(INFO) << "Only the PSO and POS permutation were loaded, SPARQL queries "
                  "with predicate variables will therefore not work"
@@ -1377,11 +1467,10 @@ vector<float> IndexImpl::getMultiplicities(
 vector<float> IndexImpl::getMultiplicities(
     Permutation::Enum permutation) const {
   const auto& p = getPermutation(permutation);
-  auto numTriples = static_cast<float>(this->numTriples().normalAndInternal_());
-  std::array<float, 3> m{
-      numTriples / numDistinctSubjects().normalAndInternal_(),
-      numTriples / numDistinctPredicates().normalAndInternal_(),
-      numTriples / numDistinctObjects().normalAndInternal_()};
+  auto numTriples = static_cast<float>(this->numTriples().normal);
+  std::array<float, 3> m{numTriples / numDistinctSubjects().normal,
+                         numTriples / numDistinctPredicates().normal,
+                         numTriples / numDistinctObjects().normal};
   return {m[p.keyOrder()[0]], m[p.keyOrder()[1]], m[p.keyOrder()[2]]};
 }
 
@@ -1424,42 +1513,39 @@ namespace {
 
 // Return a lambda that is called repeatedly with triples that are sorted by the
 // `idx`-th column and counts the number of distinct entities that occur in a
-// triple where none of the elements fulfills the `isQleverInternalTriple`
-// predicate. This is used to count the number of distinct subjects, objects,
+// triple. This is used to count the number of distinct subjects, objects,
 // and predicates during the index building.
 template <size_t idx>
-constexpr auto makeNumDistinctIdsCounter =
-    [](size_t& numDistinctIds, auto isTripleInternal) {
-      return [lastId = std::optional<Id>{}, &numDistinctIds,
-              isTripleInternal =
-                  std::move(isTripleInternal)](const auto& triple) mutable {
-        const auto& id = triple[idx];
-        if (id != lastId && !isTripleInternal(triple)) {
-          numDistinctIds++;
-          lastId = id;
-        }
-      };
-    };
+constexpr auto makeNumDistinctIdsCounter = [](size_t& numDistinctIds) {
+  return [lastId = std::optional<Id>{},
+          &numDistinctIds](const auto& triple) mutable {
+    const auto& id = triple[idx];
+    if (id != lastId) {
+      numDistinctIds++;
+      lastId = id;
+    }
+  };
+};
 }  // namespace
 
 // _____________________________________________________________________________
 template <typename... NextSorter>
 requires(sizeof...(NextSorter) <= 1)
-void IndexImpl::createPSOAndPOS(size_t numColumns, auto& isInternalTriple,
-                                BlocksOfTriples sortedTriples,
-                                NextSorter&&... nextSorter)
+void IndexImpl::createPSOAndPOSImpl(size_t numColumns,
+                                    BlocksOfTriples sortedTriples,
+                                    bool doWriteConfiguration,
+                                    NextSorter&&... nextSorter)
 
 {
   size_t numTriplesNormal = 0;
   size_t numTriplesTotal = 0;
-  auto countTriplesNormal = [&numTriplesNormal, &numTriplesTotal,
-                             &isInternalTriple](const auto& triple) mutable {
+  auto countTriplesNormal = [&numTriplesNormal, &numTriplesTotal](
+                                [[maybe_unused]] const auto& triple) mutable {
     ++numTriplesTotal;
-    numTriplesNormal += static_cast<size_t>(!isInternalTriple(triple));
+    ++numTriplesNormal;
   };
   size_t numPredicatesNormal = 0;
-  auto predicateCounter =
-      makeNumDistinctIdsCounter<1>(numPredicatesNormal, isInternalTriple);
+  auto predicateCounter = makeNumDistinctIdsCounter<1>(numPredicatesNormal);
   size_t numPredicatesTotal =
       createPermutationPair(numColumns, AD_FWD(sortedTriples), pso_, pos_,
                             nextSorter.makePushCallback()...,
@@ -1469,19 +1555,30 @@ void IndexImpl::createPSOAndPOS(size_t numColumns, auto& isInternalTriple,
                                                numPredicatesTotal);
   configurationJson_["num-triples"] = NumNormalAndInternal::fromNormalAndTotal(
       numTriplesNormal, numTriplesTotal);
-  writeConfiguration();
+  if (doWriteConfiguration) {
+    writeConfiguration();
+  }
 };
 
+// _____________________________________________________________________________
+template <typename... NextSorter>
+requires(sizeof...(NextSorter) <= 1)
+void IndexImpl::createPSOAndPOS(size_t numColumns,
+                                BlocksOfTriples sortedTriples,
+                                NextSorter&&... nextSorter) {
+  createPSOAndPOSImpl(numColumns, std::move(sortedTriples), true,
+                      AD_FWD(nextSorter)...);
+}
+
 // _____________________________________________________________________________
 template <typename... NextSorter>
 requires(sizeof...(NextSorter) <= 1)
 std::optional<PatternCreator::TripleSorter> IndexImpl::createSPOAndSOP(
-    size_t numColumns, auto& isInternalTriple, BlocksOfTriples sortedTriples,
+    size_t numColumns, BlocksOfTriples sortedTriples,
     NextSorter&&... nextSorter) {
   size_t numSubjectsNormal = 0;
   size_t numSubjectsTotal = 0;
-  auto numSubjectCounter =
-      makeNumDistinctIdsCounter<0>(numSubjectsNormal, isInternalTriple);
+  auto numSubjectCounter = makeNumDistinctIdsCounter<0>(numSubjectsNormal);
   std::optional<PatternCreator::TripleSorter> result;
   if (usePatterns_) {
     // We will return the next sorter.
@@ -1492,9 +1589,8 @@ std::optional<PatternCreator::TripleSorter> IndexImpl::createSPOAndSOP(
         onDiskBase_ + ".index.patterns",
         idOfHasPatternDuringIndexBuilding_.value(),
         memoryLimitIndexBuilding() / NUM_EXTERNAL_SORTERS_AT_SAME_TIME};
-    auto pushTripleToPatterns = [&patternCreator,
-                                 &isInternalTriple](const auto& triple) {
-      bool ignoreForPatterns = isInternalTriple(triple);
+    auto pushTripleToPatterns = [&patternCreator](const auto& triple) {
+      bool ignoreForPatterns = false;
       static_assert(NumColumnsIndexBuilding == 4,
                     "this place probably has to be changed when additional "
                     "payload columns are added");
@@ -1529,14 +1625,13 @@ std::optional<PatternCreator::TripleSorter> IndexImpl::createSPOAndSOP(
 // _____________________________________________________________________________
 template <typename... NextSorter>
 requires(sizeof...(NextSorter) <= 1)
-void IndexImpl::createOSPAndOPS(size_t numColumns, auto& isInternalTriple,
+void IndexImpl::createOSPAndOPS(size_t numColumns,
                                 BlocksOfTriples sortedTriples,
                                 NextSorter&&... nextSorter) {
   // For the last pair of permutations we don't need a next sorter, so we
   // have no fourth argument.
   size_t numObjectsNormal = 0;
-  auto objectCounter =
-      makeNumDistinctIdsCounter<2>(numObjectsNormal, isInternalTriple);
+  auto objectCounter = makeNumDistinctIdsCounter<2>(numObjectsNormal);
   size_t numObjectsTotal = createPermutationPair(
       numColumns, AD_FWD(sortedTriples), osp_, ops_,
       nextSorter.makePushCallback()..., std::ref(objectCounter));
diff --git a/src/index/IndexImpl.h b/src/index/IndexImpl.h
index ad5d73a83..de108ab66 100644
--- a/src/index/IndexImpl.h
+++ b/src/index/IndexImpl.h
@@ -78,11 +78,19 @@ struct IndexBuilderDataAsStxxlVector : IndexBuilderDataBase {
   std::vector<size_t> actualPartialSizes;
 };
 
+// Store the "normal" triples sorted by the first permutation, together with
+// the additional "internal" triples, sorted by PSO.
+struct FirstPermutationSorterAndInternalTriplesAsPso {
+  using SorterPtr =
+      std::unique_ptr<ad_utility::CompressedExternalIdTableSorterTypeErased>;
+  SorterPtr firstPermutationSorter_;
+  std::unique_ptr<ExternalSorter<SortByPSO, NumColumnsIndexBuilding>>
+      internalTriplesPso_;
+};
 // All the data from IndexBuilderDataBase and a ExternalSorter that stores all
 // ID triples sorted by the first permutation.
 struct IndexBuilderDataAsFirstPermutationSorter : IndexBuilderDataBase {
-  using SorterPtr =
-      std::unique_ptr<ad_utility::CompressedExternalIdTableSorterTypeErased>;
+  using SorterPtr = FirstPermutationSorterAndInternalTriplesAsPso;
   SorterPtr sorter_;
   IndexBuilderDataAsFirstPermutationSorter(const IndexBuilderDataBase& base,
                                            SorterPtr sorter)
@@ -469,10 +477,9 @@ class IndexImpl {
   std::unique_ptr<RdfParserBase> makeRdfParser(const std::string& filename,
                                                Index::Filetype type) const;
 
-  std::unique_ptr<ad_utility::CompressedExternalIdTableSorterTypeErased>
-  convertPartialToGlobalIds(TripleVec& data,
-                            const vector<size_t>& actualLinesPerPartial,
-                            size_t linesPerPartial);
+  FirstPermutationSorterAndInternalTriplesAsPso convertPartialToGlobalIds(
+      TripleVec& data, const vector<size_t>& actualLinesPerPartial,
+      size_t linesPerPartial, auto isQLeverInternalTriple);
 
   // Generator that returns all words in the given context file (if not empty)
   // and then all words in all literals (if second argument is true).
@@ -647,97 +654,10 @@ class IndexImpl {
   // the input).
   NumNormalAndInternal numTriples() const;
 
-  // The index contains several triples that are not part of the "actual"
-  // knowledge graph, but are added by QLever for internal reasons (e.g. for an
-  // efficient implementation of language filters). For a given
-  // `Permutation::Enum`, returns the following `std::pair`:
-  //
-  // first:  A `vector<pair<Id, Id>>` that denotes ranges in the first column
-  //         of the permutation that imply that a triple is added. For example
-  //         in the `SPO` and `SOP` permutation a literal subject means that the
-  //         triple was added (literals are not legal subjects in RDF), so the
-  //         pair `(idOfFirstLiteral, idOfLastLiteral + 1)` will be contained
-  //         in the vector.
-  // second: A lambda that checks for a triple *that is not already excluded
-  //         by the ignored ranges from the first argument* whether it still
-  //         is an added triple. For example in the `Sxx` and `Oxx` permutation
-  //         a triple where the predicate starts with '@' (instead of the usual
-  //         '<' is an added triple from the language filter implementation.
-  //
-  // Note: A triple from a given permutation is an added triple if and only if
-  //       it's first column is contained in any of the ranges from `first` OR
-  //       the lambda `second` returns true for that triple.
-  //
-  // For example usages see `IndexScan.cpp` (the implementation of the full
-  // index scan) and `GroupBy.cpp`.
-  auto getIgnoredIdRanges(const Permutation::Enum permutation) const {
-    std::vector<std::pair<Id, Id>> ignoredRanges;
-    ignoredRanges.emplace_back(qlever::getBoundsForSpecialIds());
-
-    auto literalRanges =
-        getVocab().prefixRanges(ad_utility::triple_component::literalPrefix);
-    auto taggedPredicatesRanges =
-        getVocab().prefixRanges(ad_utility::languageTaggedPredicatePrefix);
-    std::string internal{INTERNAL_ENTITIES_URI_PREFIX};
-    internal[0] = ad_utility::triple_component::iriPrefixChar;
-    auto internalEntitiesRanges = getVocab().prefixRanges(internal);
-
-    auto pushIgnoredRange = [&ignoredRanges](const auto& ranges) {
-      for (const auto& range : ranges.ranges()) {
-        ignoredRanges.emplace_back(Id::makeFromVocabIndex(range.first),
-                                   Id::makeFromVocabIndex(range.second));
-      }
-    };
-    pushIgnoredRange(internalEntitiesRanges);
-    using enum Permutation::Enum;
-    if (permutation == SPO || permutation == SOP) {
-      pushIgnoredRange(literalRanges);
-    } else if (permutation == PSO || permutation == POS) {
-      pushIgnoredRange(taggedPredicatesRanges);
-    }
-
-    // A lambda that checks whether the `predicateId` is an internal ID like
-    // `ql:has-pattern` or `@en@rdfs:label`.
-    auto isInternalPredicateId = [internalEntitiesRanges,
-                                  taggedPredicatesRanges](Id predicateId) {
-      if (predicateId.getDatatype() == Datatype::Undefined) {
-        return true;
-      }
-      AD_CORRECTNESS_CHECK(predicateId.getDatatype() == Datatype::VocabIndex);
-      auto index = predicateId.getVocabIndex();
-      return (internalEntitiesRanges.contain(index) ||
-              taggedPredicatesRanges.contain(index));
-    };
-
-    auto isTripleIgnored = [permutation,
-                            isInternalPredicateId](const auto& triple) {
-      // TODO<joka921, everybody in the future>:
-      // A lot of code (especially for statistical queries in `GroupBy.cpp` and
-      // the pattern trick) relies on this function being a noop for the `PSO`
-      // and `POS` permutations, meaning that it suffices to check the
-      // `ignoredRanges` for them. Should this ever change (which means that we
-      // add internal triples that use predicates that are actually contained in
-      // the knowledge graph), then all the code that uses this function has to
-      // be thoroughly reviewed.
-      if (permutation == SPO || permutation == OPS) {
-        // Predicates are always entities from the vocabulary.
-        return isInternalPredicateId(triple[1]);
-      } else if (permutation == SOP || permutation == OSP) {
-        return isInternalPredicateId(triple[2]);
-      }
-      return false;
-    };
-
-    return std::pair{std::move(ignoredRanges), std::move(isTripleIgnored)};
-  }
   using BlocksOfTriples = cppcoro::generator<IdTableStatic<0>>;
 
   // Functions to create the pairs of permutations during the index build. Each
   // of them takes the following arguments:
-  // * `isQleverInternalTriple` a callable that takes an `Id` and returns true
-  // iff
-  //    the corresponding IRI was internally added by QLever and not part of the
-  //    knowledge graph.
   // * `sortedInput`  The input, must be sorted by the first permutation in the
   //    function name.
   // * `nextSorter` A callback that is invoked for each row in each of the
@@ -750,25 +670,37 @@ class IndexImpl {
   template <typename... NextSorter>
   requires(sizeof...(NextSorter) <= 1)
   std::optional<PatternCreator::TripleSorter> createSPOAndSOP(
-      size_t numColumns, auto& isInternalTriple, BlocksOfTriples sortedTriples,
+      size_t numColumns, BlocksOfTriples sortedTriples,
       NextSorter&&... nextSorter);
   // Create the OSP and OPS permutations. Additionally, count the number of
   // distinct objects and write it to the metadata.
   template <typename... NextSorter>
   requires(sizeof...(NextSorter) <= 1)
-  void createOSPAndOPS(size_t numColumns, auto& isInternalTriple,
-                       BlocksOfTriples sortedTriples,
+  void createOSPAndOPS(size_t numColumns, BlocksOfTriples sortedTriples,
                        NextSorter&&... nextSorter);
 
   // Create the PSO and POS permutations. Additionally, count the number of
   // distinct predicates and the number of actual triples and write them to the
-  // metadata.
+  // metadata. The meta-data JSON file for the index statistics will only be
+  // written iff `doWriteConfiguration` is true. That parameter is set to
+  // `false` when building the additional permutations for the internal triples.
   template <typename... NextSorter>
   requires(sizeof...(NextSorter) <= 1)
-  void createPSOAndPOS(size_t numColumns, auto& isInternalTriple,
-                       BlocksOfTriples sortedTriples,
+  void createPSOAndPOSImpl(size_t numColumns, BlocksOfTriples sortedTriples,
+                           bool doWriteConfiguration,
+                           NextSorter&&... nextSorter);
+  // Call `createPSOAndPOSImpl` with the given arguments and with
+  // `doWriteConfiguration` set to `true` (see above).
+  template <typename... NextSorter>
+  requires(sizeof...(NextSorter) <= 1)
+  void createPSOAndPOS(size_t numColumns, BlocksOfTriples sortedTriples,
                        NextSorter&&... nextSorter);
 
+  // Create the internal PSO and POS permutations from the sorted internal
+  // triples. Return `(numInternalTriples, numInternalPredicates)`.
+  std::pair<size_t, size_t> createInternalPSOandPOS(
+      auto&& internalTriplesPsoSorter);
+
   // Set up one of the permutation sorters with the appropriate memory limit.
   // The `permutationName` is used to determine the filename and must be unique
   // for each call during one index build.
@@ -822,5 +754,12 @@ class IndexImpl {
   // PSO and POS permutations.
   std::unique_ptr<ExternalSorter<SortByPSO, NumColumnsIndexBuilding + 2>>
   buildOspWithPatterns(PatternCreator::TripleSorter sortersFromPatternCreator,
-                       auto isQleverInternalTriple);
+                       auto& internalTripleSorter);
+
+  // During the index, building add the number of internal triples and internal
+  // predicates to the configuration. Note: the number of internal objects and
+  // subjects will always remain zero, because we have no cheap way of computing
+  // them.
+  void addInternalStatisticsToConfiguration(size_t numTriplesInternal,
+                                            size_t numPredicatesInternal);
 };
diff --git a/src/engine/LocalVocabEntry.cpp b/src/index/LocalVocabEntry.cpp
similarity index 100%
rename from src/engine/LocalVocabEntry.cpp
rename to src/index/LocalVocabEntry.cpp
diff --git a/src/engine/LocalVocabEntry.h b/src/index/LocalVocabEntry.h
similarity index 100%
rename from src/engine/LocalVocabEntry.h
rename to src/index/LocalVocabEntry.h
diff --git a/src/index/Permutation.cpp b/src/index/Permutation.cpp
index a126a50f9..daa1000ec 100644
--- a/src/index/Permutation.cpp
+++ b/src/index/Permutation.cpp
@@ -5,6 +5,7 @@
 #include "index/Permutation.h"
 
 #include "absl/strings/str_cat.h"
+#include "index/ConstantsIndexBuilding.h"
 #include "util/StringUtils.h"
 
 // _____________________________________________________________________
@@ -12,10 +13,20 @@ Permutation::Permutation(Enum permutation, Allocator allocator)
     : readableName_(toString(permutation)),
       fileSuffix_(absl::StrCat(".", ad_utility::utf8ToLower(readableName_))),
       keyOrder_(toKeyOrder(permutation)),
-      allocator_{std::move(allocator)} {}
+      allocator_{std::move(allocator)},
+      permutation_{permutation} {}
 
 // _____________________________________________________________________
-void Permutation::loadFromDisk(const std::string& onDiskBase) {
+void Permutation::loadFromDisk(const std::string& onDiskBase,
+                               std::function<bool(Id)> isInternalId,
+                               bool loadInternalPermutation) {
+  isInternalId_ = std::move(isInternalId);
+  if (loadInternalPermutation) {
+    internalPermutation_ =
+        std::make_unique<Permutation>(permutation_, allocator_);
+    internalPermutation_->loadFromDisk(
+        absl::StrCat(onDiskBase, INTERNAL_INDEX_INFIX), isInternalId_, false);
+  }
   if constexpr (MetaData::isMmapBased_) {
     meta_.setup(onDiskBase + ".index" + fileSuffix_ + MMAP_FILE_SUFFIX,
                 ad_utility::ReuseTag(), ad_utility::AccessPattern::Random);
@@ -48,21 +59,25 @@ IdTable Permutation::scan(const ScanSpecification& scanSpec,
                              readableName_ + ", which was not loaded");
   }
 
-  return reader().scan(scanSpec, meta_.blockData(), additionalColumns,
-                       cancellationHandle, limitOffset);
+  const auto& p = getActualPermutation(scanSpec);
+
+  return p.reader().scan(scanSpec, p.meta_.blockData(), additionalColumns,
+                         cancellationHandle, limitOffset);
 }
 
 // _____________________________________________________________________
 size_t Permutation::getResultSizeOfScan(
     const ScanSpecification& scanSpec) const {
-  return reader().getResultSizeOfScan(scanSpec, meta_.blockData());
+  const auto& p = getActualPermutation(scanSpec);
+  return p.reader().getResultSizeOfScan(scanSpec, p.meta_.blockData());
 }
 
 // ____________________________________________________________________________
 IdTable Permutation::getDistinctCol1IdsAndCounts(
     Id col0Id, const CancellationHandle& cancellationHandle) const {
-  return reader().getDistinctCol1IdsAndCounts(col0Id, meta_.blockData(),
-                                              cancellationHandle);
+  const auto& p = getActualPermutation(col0Id);
+  return p.reader().getDistinctCol1IdsAndCounts(col0Id, p.meta_.blockData(),
+                                                cancellationHandle);
 }
 
 // ____________________________________________________________________________
@@ -115,20 +130,22 @@ std::string_view Permutation::toString(Permutation::Enum permutation) {
 // _____________________________________________________________________
 std::optional<CompressedRelationMetadata> Permutation::getMetadata(
     Id col0Id) const {
-  if (meta_.col0IdExists(col0Id)) {
-    return meta_.getMetaData(col0Id);
+  const auto& p = getActualPermutation(col0Id);
+  if (p.meta_.col0IdExists(col0Id)) {
+    return p.meta_.getMetaData(col0Id);
   }
-  return reader().getMetadataForSmallRelation(meta_.blockData(), col0Id);
+  return p.reader().getMetadataForSmallRelation(p.meta_.blockData(), col0Id);
 }
 
 // _____________________________________________________________________
 std::optional<Permutation::MetadataAndBlocks> Permutation::getMetadataAndBlocks(
     const ScanSpecification& scanSpec) const {
+  const auto& p = getActualPermutation(scanSpec);
   CompressedRelationReader::ScanSpecAndBlocks mb{
-      scanSpec,
-      CompressedRelationReader::getRelevantBlocks(scanSpec, meta_.blockData())};
+      scanSpec, CompressedRelationReader::getRelevantBlocks(
+                    scanSpec, p.meta_.blockData())};
 
-  auto firstAndLastTriple = reader().getFirstAndLastTriple(mb);
+  auto firstAndLastTriple = p.reader().getFirstAndLastTriple(mb);
   if (!firstAndLastTriple.has_value()) {
     return std::nullopt;
   }
@@ -143,13 +160,40 @@ Permutation::IdTableGenerator Permutation::lazyScan(
     ColumnIndicesRef additionalColumns,
     ad_utility::SharedCancellationHandle cancellationHandle,
     const LimitOffsetClause& limitOffset) const {
+  const auto& p = getActualPermutation(scanSpec);
   if (!blocks.has_value()) {
     auto blockSpan = CompressedRelationReader::getRelevantBlocks(
-        scanSpec, meta_.blockData());
+        scanSpec, p.meta_.blockData());
     blocks = std::vector(blockSpan.begin(), blockSpan.end());
   }
   ColumnIndices columns{additionalColumns.begin(), additionalColumns.end()};
-  return reader().lazyScan(scanSpec, std::move(blocks.value()),
-                           std::move(columns), std::move(cancellationHandle),
-                           limitOffset);
+  return p.reader().lazyScan(scanSpec, std::move(blocks.value()),
+                             std::move(columns), std::move(cancellationHandle),
+                             limitOffset);
+}
+
+// ______________________________________________________________________
+const Permutation& Permutation::getActualPermutation(
+    const ScanSpecification& spec) const {
+  auto isInternal = [this](const std::optional<Id>& id) {
+    return id.has_value() && isInternalId_(id.value());
+  };
+
+  bool isInternalScan = isInternal(spec.col0Id()) ||
+                        isInternal(spec.col1Id()) || isInternal(spec.col2Id());
+
+  if (!isInternalScan) {
+    return *this;
+  }
+  AD_CORRECTNESS_CHECK(internalPermutation_ != nullptr, [this]() {
+    return absl::StrCat("No internal triples were loaded for the permutation ",
+                        readableName_);
+  });
+  return *internalPermutation_;
+}
+
+// ______________________________________________________________________
+const Permutation& Permutation::getActualPermutation(Id id) const {
+  return getActualPermutation(
+      ScanSpecification{id, std::nullopt, std::nullopt});
 }
diff --git a/src/index/Permutation.h b/src/index/Permutation.h
index 6cbd05f82..c10337291 100644
--- a/src/index/Permutation.h
+++ b/src/index/Permutation.h
@@ -51,7 +51,9 @@ class Permutation {
   explicit Permutation(Enum permutation, Allocator allocator);
 
   // everything that has to be done when reading an index from disk
-  void loadFromDisk(const std::string& onDiskBase);
+  void loadFromDisk(const std::string& onDiskBase,
+                    std::function<bool(Id)> isInternalId,
+                    bool loadAdditional = false);
 
   // For a given ID for the col0, retrieve all IDs of the col1 and col2.
   // If `col1Id` is specified, only the col2 is returned for triples that
@@ -132,6 +134,10 @@ class Permutation {
   // _______________________________________________________
   const MetaData& metaData() const { return meta_; }
 
+  // _______________________________________________________
+  const Permutation& getActualPermutation(const ScanSpecification& spec) const;
+  const Permutation& getActualPermutation(Id id) const;
+
  private:
   // for Log output, e.g. "POS"
   std::string readableName_;
@@ -149,4 +155,9 @@ class Permutation {
   Allocator allocator_;
 
   bool isLoaded_ = false;
+
+  Enum permutation_;
+  std::unique_ptr<Permutation> internalPermutation_ = nullptr;
+
+  std::function<bool(Id)> isInternalId_;
 };
diff --git a/src/index/PermutationExporterMain.cpp b/src/index/PermutationExporterMain.cpp
deleted file mode 100644
index b95fea6b5..000000000
--- a/src/index/PermutationExporterMain.cpp
+++ /dev/null
@@ -1,87 +0,0 @@
-//  Copyright 2021, University of Freiburg, Chair of Algorithms and Data
-//  Structures. Author: Johannes Kalmbach <kalmbacj@cs.uni-freiburg.de>
-
-#include "./Index.h"
-#include "./IndexImpl.h"
-#include "./TriplesView.h"
-#include "util/MemorySize/MemorySize.h"
-
-/// Dump a  certain permutation to stdout in a human-readable way as IDs, and
-/// in deterministic order
-template <typename Permutation>
-void dumpToStdout(const Permutation& permutation) {
-  ad_utility::AllocatorWithLimit<Id> allocator{
-      ad_utility::makeAllocationMemoryLeftThreadsafeObject(
-          ad_utility::MemorySize::max())};
-  auto triples = TriplesView(
-      permutation, std::make_shared<ad_utility::CancellationHandle<>>());
-  size_t i = 0;
-  for (auto triple : triples) {
-    std::cout << triple[0] << " " << triple[1] << " " << triple[2] << std::endl;
-    ++i;
-    if (i % (1ul << 20) == 0) {
-      LOG(INFO) << "Exported " << i << " relations" << std::endl;
-    }
-  }
-}
-
-/// Load a certain permutation from a certain index, and dump it to stdout
-/// in a human-readable and deterministic way. This can be used for large
-/// regression tests, when the index format or the index building procedure
-/// changes.
-/// Args: ./PermutationExporterMain <indexBasename> <permutation>
-///  (<permutation>  must be one of pso, pos, spo, sop, osp, ops
-int main(int argc, char** argv) {
-  // Actual output goes to std::cout,output of LOG(...) to std::cerr
-  ad_utility::setGlobalLoggingStream(&std::cerr);
-
-  if (argc != 3) {
-    LOG(ERROR) << "Usage: PermutationExporterMain <indexBasename> <permutation "
-                  "to dump>"
-               << std::endl;
-    return EXIT_FAILURE;
-  }
-
-  Index i{ad_utility::makeUnlimitedAllocator<Id>()};
-  IndexImpl& impl = i.getImpl();
-  std::string indexName{argv[1]};
-  std::string p{argv[2]};
-
-  if (p == "sop") {
-    impl.SOP().loadFromDisk(indexName);
-    dumpToStdout(impl.SOP());
-    return EXIT_SUCCESS;
-  }
-  if (p == "spo") {
-    impl.SPO().loadFromDisk(indexName);
-    dumpToStdout(impl.SPO());
-    return EXIT_SUCCESS;
-  }
-  if (p == "osp") {
-    impl.OSP().loadFromDisk(indexName);
-    dumpToStdout(impl.OSP());
-    return EXIT_SUCCESS;
-  }
-  if (p == "ops") {
-    impl.OPS().loadFromDisk(indexName);
-    dumpToStdout(impl.OPS());
-    return EXIT_SUCCESS;
-  }
-
-  if (p == "pos") {
-    impl.POS().loadFromDisk(indexName);
-    dumpToStdout(impl.POS());
-    return EXIT_SUCCESS;
-  }
-
-  if (p == "pso") {
-    impl.PSO().loadFromDisk(indexName);
-    dumpToStdout(impl.PSO());
-    return EXIT_SUCCESS;
-  }
-
-  LOG(ERROR)
-      << "<permutation> must be one of pso, pos, spo, sop, osp, ops, but was \""
-      << p << "\"" << std::endl;
-  return EXIT_FAILURE;
-}
diff --git a/src/index/TriplesView.h b/src/index/TriplesView.h
deleted file mode 100644
index 3b08c9be2..000000000
--- a/src/index/TriplesView.h
+++ /dev/null
@@ -1,77 +0,0 @@
-#pragma once
-
-#include <utility>
-#include <vector>
-
-#include "CompressedRelation.h"
-#include "engine/idTable/IdTable.h"
-#include "global/Id.h"
-#include "util/AllocatorWithLimit.h"
-#include "util/CancellationHandle.h"
-#include "util/File.h"
-#include "util/Generator.h"
-
-namespace detail {
-using IgnoredRelations = std::vector<std::pair<Id, Id>>;
-inline auto alwaysReturnFalse = [](auto&&...) { return false; };
-}  // namespace detail
-
-/**
- * @brief Yield all triples from a given permutations
- * @param permutation
- * @param allocator The current implementation reads complete relations into
- *        memory. The `allocator` is used  to allocate the needed buffers for
- *        this.
- * @param ignoredRanges For each range (a, b) in `ignoredRanges` a triple
- *        will not be yielded if a <= triple[0] < b. Specifying contiguous
- *        ranges of ignored relations is more efficient than the
- *        `isTripleIgnored` callable (see below), because the ignored relations
- *        will never be read from disk. If the different pairs overlap, the
- *        behavior is undefined.
- * @param isTripleIgnored For each triple `isTripleIgnored(triple)` is called.
- *        The triple is only yielded, if the result of this call is `false` ̇.
- */
-template <typename IsTripleIgnored = decltype(detail::alwaysReturnFalse)>
-cppcoro::generator<std::array<Id, 3>> TriplesView(
-    const auto& permutation,
-    ad_utility::SharedCancellationHandle cancellationHandle,
-    detail::IgnoredRelations ignoredRanges = {},
-    IsTripleIgnored isTripleIgnored = IsTripleIgnored{}) {
-  std::ranges::sort(ignoredRanges);
-
-  auto blockGenerator = permutation.lazyScan(
-      {std::nullopt, std::nullopt, std::nullopt}, std::nullopt,
-      CompressedRelationReader::ColumnIndices{}, cancellationHandle);
-  auto ignoreIt = ignoredRanges.begin();
-  Id ignoreLow = Id::max();
-  Id ignoreHigh = Id::max();
-  if (ignoreIt != ignoredRanges.end()) {
-    std::tie(ignoreLow, ignoreHigh) = *ignoreIt;
-    ++ignoreIt;
-  }
-  for (const IdTable& cols : blockGenerator) {
-    AD_CORRECTNESS_CHECK(cols.numColumns() == 3);
-    for (const auto& row : cols) {
-      // TODO<joka921> This can be done much more efficient without copying etc.
-      // and with prefiltering.
-      std::array<Id, 3> triple{row[0], row[1], row[2]};
-      if (isTripleIgnored(triple)) [[unlikely]] {
-        continue;
-      }
-      if (triple[0] > ignoreHigh) {
-        if (ignoreIt != ignoredRanges.end()) {
-          std::tie(ignoreLow, ignoreHigh) = *ignoreIt;
-          ++ignoreIt;
-        } else {
-          ignoreLow = Id::max();
-          ignoreHigh = Id::max();
-        }
-      }
-      if (triple[0] >= ignoreLow && triple[0] < ignoreHigh) {
-        continue;
-      }
-      co_yield triple;
-    }
-    cancellationHandle->throwIfCancelled();
-  }
-}


Test patch:
diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt
index a47c2f592..02ce2123c 100644
--- a/test/CMakeLists.txt
+++ b/test/CMakeLists.txt
@@ -237,8 +237,6 @@ addLinkAndDiscoverTest(CompressorStreamTest engine)
 
 addLinkAndDiscoverTest(AsyncStreamTest)
 
-addLinkAndDiscoverTest(TriplesViewTest util OpenSSL::SSL OpenSSL::Crypto)
-
 addLinkAndDiscoverTest(BitUtilsTest)
 
 addLinkAndDiscoverTest(NBitIntegerTest)
@@ -407,7 +405,7 @@ addLinkAndDiscoverTest(FilterTest engine)
 
 addLinkAndDiscoverTest(ResultTest engine)
 
-addLinkAndDiscoverTest(SparqlExpressionGeneratorsTest)
+addLinkAndDiscoverTest(SparqlExpressionGeneratorsTest engine)
 
 addLinkAndDiscoverTest(UrlParserTest)
 
diff --git a/test/GroupByTest.cpp b/test/GroupByTest.cpp
index ab402204a..0fc1b0dcd 100644
--- a/test/GroupByTest.cpp
+++ b/test/GroupByTest.cpp
@@ -1728,13 +1728,8 @@ TEST_F(GroupByOptimizations, computeGroupByForFullIndexScan) {
         ::testing::ElementsAre(getId("<a>"), getId("<b>"), getId("<c>"),
                                getId("<x>"), getId("<y>"), getId("<z>")));
     if (includeCount) {
-      EXPECT_THAT(
-          result.getColumn(1),
-          ::testing::ElementsAre(I(2), I(2), I(2), I(7), I(1),
-                                 // TODO<joka921> This should be 1.
-                                 // There is one triple added <z> @en@<label>
-                                 // "zz"@en which is currently not filtered out.
-                                 I(2)));
+      EXPECT_THAT(result.getColumn(1),
+                  ::testing::ElementsAre(I(2), I(2), I(2), I(7), I(1), I(1)));
     }
   };
   testWithBothInterfaces(true, true);
diff --git a/test/IndexTest.cpp b/test/IndexTest.cpp
index 866072aa9..0d45c4aec 100644
--- a/test/IndexTest.cpp
+++ b/test/IndexTest.cpp
@@ -400,100 +400,6 @@ TEST(IndexTest, TripleToInternalRepresentation) {
   }
 }
 
-TEST(IndexTest, getIgnoredIdRanges) {
-  const Index& indexNoImpl = getQec()->getIndex();
-  const IndexImpl& index = indexNoImpl.getImpl();
-
-  // First manually get the IDs of the vocabulary elements that might appear
-  // in added triples.
-  auto getId = makeGetId(indexNoImpl);
-  Id label = getId("<label>");
-  Id firstLiteral = getId("\"A\"");
-  Id lastLiteral = getId("\"zz\"@en");
-  Id x = getId("<x>");
-  Id enLabel =
-      getId(ad_utility::convertToLanguageTaggedPredicate("<label>", "en"));
-  Id qlLangtag =
-      getId("<http://qlever.cs.uni-freiburg.de/builtin-functions/langtag>");
-  Id en = getId("<http://qlever.cs.uni-freiburg.de/builtin-functions/@en>");
-
-  auto increment = [](Id id) {
-    return Id::makeFromVocabIndex(id.getVocabIndex().incremented());
-  };
-
-  // The range of all entities that start with
-  // "<http://qlever.cs.uni-freiburg.de/builtin-functions/"
-  auto internalEntities = std::pair{en, increment(qlLangtag)};
-
-  // The range of all entities that start with @ (like `@en@<label>`)
-  auto predicatesWithLangtag = std::pair{enLabel, increment(enLabel)};
-  // The range of all literals;
-  auto literals = std::pair{firstLiteral, increment(lastLiteral)};
-  // Nothing in the external vocabulary for this test.
-  //
-
-  auto specialIds = qlever::getBoundsForSpecialIds();
-  {
-    auto [ranges, lambda] = index.getIgnoredIdRanges(Permutation::POS);
-    ASSERT_FALSE(lambda(std::array{label, firstLiteral, x}));
-
-    // Note: The following triple is added, but it should be filtered out via
-    // the ranges and not via the lambda, because it can be retrieved using the
-    // `ranges`.
-    ASSERT_FALSE(lambda(std::array{enLabel, firstLiteral, x}));
-    ASSERT_FALSE(lambda(std::array{x, x, x}));
-    EXPECT_THAT(ranges,
-                UnorderedElementsAre(internalEntities, predicatesWithLangtag,
-                                     specialIds));
-  }
-  {
-    auto [ranges, lambda] = index.getIgnoredIdRanges(Permutation::PSO);
-    ASSERT_FALSE(lambda(std::array{label, x, firstLiteral}));
-
-    // Note: The following triple is added, but it should be filtered out via
-    // the ranges and not via the lambda, because it can be retrieved using the
-    // `ranges`.
-    ASSERT_FALSE(lambda(std::array{enLabel, x, firstLiteral}));
-    ASSERT_FALSE(lambda(std::array{x, x, x}));
-    EXPECT_THAT(ranges,
-                UnorderedElementsAre(internalEntities, predicatesWithLangtag,
-                                     specialIds));
-  }
-  {
-    auto [ranges, lambda] = index.getIgnoredIdRanges(Permutation::SOP);
-    ASSERT_TRUE(lambda(std::array{x, firstLiteral, enLabel}));
-    ASSERT_FALSE(lambda(std::array{x, firstLiteral, label}));
-    ASSERT_FALSE(lambda(std::array{x, x, label}));
-    EXPECT_THAT(ranges,
-                UnorderedElementsAre(internalEntities, literals, specialIds));
-  }
-  {
-    auto [ranges, lambda] = index.getIgnoredIdRanges(Permutation::SPO);
-    ASSERT_TRUE(lambda(std::array{x, enLabel, firstLiteral}));
-    ASSERT_FALSE(lambda(std::array{x, label, firstLiteral}));
-    ASSERT_FALSE(lambda(std::array{x, label, x}));
-    EXPECT_THAT(ranges,
-                UnorderedElementsAre(internalEntities, literals, specialIds));
-  }
-  {
-    auto [ranges, lambda] = index.getIgnoredIdRanges(Permutation::OSP);
-    ASSERT_TRUE(lambda(std::array{firstLiteral, x, enLabel}));
-    ASSERT_FALSE(lambda(std::array{firstLiteral, x, label}));
-    ASSERT_FALSE(lambda(std::array{x, x, label}));
-    EXPECT_THAT(ranges, UnorderedElementsAre(internalEntities, specialIds));
-  }
-  {
-    auto [ranges, lambda] = index.getIgnoredIdRanges(Permutation::OPS);
-    ASSERT_TRUE(lambda(std::array{firstLiteral, enLabel, x}));
-    ASSERT_FALSE(lambda(std::array{firstLiteral, label, x}));
-    ASSERT_FALSE(lambda(std::array{x, label, x}));
-    auto hasPattern =
-        qlever::specialIds().at(std::string{HAS_PATTERN_PREDICATE});
-    ASSERT_TRUE(lambda(std::array{firstLiteral, hasPattern, x}));
-    EXPECT_THAT(ranges, UnorderedElementsAre(internalEntities, specialIds));
-  }
-}
-
 TEST(IndexTest, NumDistinctEntities) {
   std::string turtleInput =
       "<x> <label> \"alpha\" . <x> <label> \"älpha\" . <x> <label> \"A\" . "
@@ -505,28 +411,23 @@ TEST(IndexTest, NumDistinctEntities) {
   // `IndexTestHelpers.cpp` change.
   // TODO<joka921> Also check the number of triples and the number of
   // added things.
-  auto subjects = index.numDistinctSubjects();
-  EXPECT_EQ(subjects.normal, 3);
-  // All literals with language tags are added subjects.
-  EXPECT_EQ(subjects.internal, 1);
-  EXPECT_EQ(subjects, index.numDistinctCol0(Permutation::SPO));
-  EXPECT_EQ(subjects, index.numDistinctCol0(Permutation::SOP));
-
-  auto predicates = index.numDistinctPredicates();
-  EXPECT_EQ(predicates.normal, 2);
-  // The added predicates are `ql:has-pattern`, `ql:langtag`, and one added
+  auto numSubjects = index.numDistinctSubjects();
+  EXPECT_EQ(numSubjects.normal, 3);
+  // All literals with language tags are added numSubjects.
+  EXPECT_EQ(numSubjects, index.numDistinctCol0(Permutation::SPO));
+  EXPECT_EQ(numSubjects, index.numDistinctCol0(Permutation::SOP));
+
+  auto numPredicates = index.numDistinctPredicates();
+  EXPECT_EQ(numPredicates.normal, 2);
+  // The added numPredicates are `ql:has-pattern`, `ql:langtag`, and one added
   // predicate for each combination of predicate+language that is actually used
   // (e.g. `@en@label`).
-  EXPECT_EQ(predicates.internal, 3);
-  EXPECT_EQ(predicates, index.numDistinctCol0(Permutation::PSO));
-  EXPECT_EQ(predicates, index.numDistinctCol0(Permutation::POS));
+  EXPECT_EQ(numPredicates.internal, 3);
+  EXPECT_EQ(numPredicates, index.numDistinctCol0(Permutation::PSO));
+  EXPECT_EQ(numPredicates, index.numDistinctCol0(Permutation::POS));
 
   auto objects = index.numDistinctObjects();
   EXPECT_EQ(objects.normal, 7);
-  // One added object for each language that is used.
-  // Note: The pattern indices from the `ql:has-pattern` predicate are currently
-  // not part of `objects.internal`, but they are also not very important.
-  EXPECT_EQ(objects.internal, 1);
   EXPECT_EQ(objects, index.numDistinctCol0(Permutation::OSP));
   EXPECT_EQ(objects, index.numDistinctCol0(Permutation::OPS));
 
@@ -537,9 +438,11 @@ TEST(IndexTest, NumDistinctEntities) {
   EXPECT_EQ(numTriples.internal, 5);
 
   auto multiplicities = index.getMultiplicities(Permutation::SPO);
-  EXPECT_FLOAT_EQ(multiplicities[0], 12.0 / 4.0);
-  EXPECT_FLOAT_EQ(multiplicities[1], 12.0 / 5.0);
-  EXPECT_FLOAT_EQ(multiplicities[2], 12.0 / 8.0);
+  // 7 triples, three distinct numSubjects, 2 distinct numPredicates, 7 distinct
+  // objects.
+  EXPECT_FLOAT_EQ(multiplicities[0], 7.0 / 3.0);
+  EXPECT_FLOAT_EQ(multiplicities[1], 7.0 / 2.0);
+  EXPECT_FLOAT_EQ(multiplicities[2], 7.0 / 7.0);
 
   multiplicities = index.getMultiplicities(iri("<x>"), Permutation::SPO);
   EXPECT_FLOAT_EQ(multiplicities[0], 2.5);
diff --git a/test/TriplesViewTest.cpp b/test/TriplesViewTest.cpp
deleted file mode 100644
index 956100991..000000000
--- a/test/TriplesViewTest.cpp
+++ /dev/null
@@ -1,114 +0,0 @@
-//  Copyright 2022, University of Freiburg,
-//  Chair of Algorithms and Data Structures.
-//  Author: Johannes Kalmbach <kalmbach@cs.uni-freiburg.de>
-
-#include <gtest/gtest.h>
-
-#include "./util/AllocatorTestHelpers.h"
-#include "./util/IdTestHelpers.h"
-#include "index/TriplesView.h"
-
-namespace {
-auto V = ad_utility::testing::VocabId;
-
-// This struct mocks the structure of the actual `Permutation::Enum` types used
-// in QLever for testing the `TriplesView`.
-struct DummyPermutation {
-  IdTable scan(std::optional<Id> col0IdDummy, std::optional<Id> col1Id) const {
-    IdTable result(3, ad_utility::makeUnlimitedAllocator<Id>());
-    AD_CORRECTNESS_CHECK(!col1Id.has_value());
-    AD_CORRECTNESS_CHECK(!col0IdDummy.has_value());
-    std::vector<Id> ids{V(1), V(3), V(5), V(7), V(8), V(10), V(13)};
-    for (auto col0Id : ids) {
-      for (size_t i = 0; i < col0Id.getVocabIndex().get(); ++i) {
-        result.push_back(std::array{col0Id,
-                                    V((i + 1) * col0Id.getVocabIndex().get()),
-                                    V((i + 2) * col0Id.getVocabIndex().get())});
-      }
-    }
-    return result;
-  }
-
-  cppcoro::generator<IdTable> lazyScan(
-      ScanSpecification scanSpec,
-      std::optional<std::vector<CompressedBlockMetadata>> blocks,
-      std::span<const ColumnIndex>, const auto&) const {
-    AD_CORRECTNESS_CHECK(!blocks.has_value());
-    auto table = scan(scanSpec.col0Id(), scanSpec.col1Id());
-    co_yield table;
-  }
-
-  struct Metadata {
-    struct Data {
-      std::vector<Id> _ids{V(1), V(3), V(5), V(7), V(8), V(10), V(13)};
-      using BaseIterator = std::vector<Id>::const_iterator;
-      struct Iterator : BaseIterator {
-        Iterator(BaseIterator b) : BaseIterator{std::move(b)} {}
-        static Id getIdFromElement(Id id) { return id; }
-        Id getId() const { return *(*this); }
-      };
-      Iterator ordered_begin() const { return _ids.begin(); }
-      Iterator ordered_end() const { return _ids.end(); }
-    };
-    Data data() const { return {}; }
-  };
-
-  Metadata meta_;
-};
-
-std::vector<std::array<Id, 3>> expectedResult() {
-  std::vector<Id> ids{V(1), V(3), V(5), V(7), V(8), V(10), V(13)};
-  std::vector<std::array<Id, 3>> result;
-  for (auto idComplex : ids) {
-    auto id = idComplex.getVocabIndex().get();
-    for (size_t i = 0; i < id; ++i) {
-      result.push_back(std::array{V(id), V(id * (i + 1)), V(id * (i + 2))});
-    }
-  }
-  return result;
-}
-}  // namespace
-
-TEST(TriplesView, AllTriples) {
-  std::vector<std::array<Id, 3>> result;
-  for (auto triple :
-       TriplesView(DummyPermutation{},
-                   std::make_shared<ad_utility::CancellationHandle<>>())) {
-    result.push_back(triple);
-  }
-  ASSERT_EQ(result, expectedResult());
-}
-
-TEST(TriplesView, IgnoreRanges) {
-  std::vector<std::array<Id, 3>> result;
-  auto expected = expectedResult();
-  std::erase_if(expected, [](const auto& triple) {
-    auto t = triple[0].getVocabIndex().get();
-    return t == 1 || t == 3 || t == 7 || t == 13;
-  });
-  std::vector<std::pair<Id, Id>> ignoredRanges{
-      {V(0), V(4)}, {V(7), V(8)}, {V(13), V(87593)}};
-  for (auto triple :
-       TriplesView(DummyPermutation{},
-                   std::make_shared<ad_utility::CancellationHandle<>>(),
-                   ignoredRanges)) {
-    result.push_back(triple);
-  }
-  ASSERT_EQ(result, expected);
-}
-
-TEST(TriplesView, IgnoreTriples) {
-  std::vector<std::array<Id, 3>> result;
-  auto expected = expectedResult();
-  auto isTripleIgnored = [](const auto& triple) {
-    return triple[1].getVocabIndex().get() % 2 == 0;
-  };
-  std::erase_if(expected, isTripleIgnored);
-  for (auto triple :
-       TriplesView(DummyPermutation{},
-                   std::make_shared<ad_utility::CancellationHandle<>>(), {},
-                   isTripleIgnored)) {
-    result.push_back(triple);
-  }
-  ASSERT_EQ(result, expected);
-}
diff --git a/test/engine/IndexScanTest.cpp b/test/engine/IndexScanTest.cpp
index 8b8e3506b..255973b38 100644
--- a/test/engine/IndexScanTest.cpp
+++ b/test/engine/IndexScanTest.cpp
@@ -418,9 +418,7 @@ TEST(IndexScan, getResultSizeOfScan) {
   {
     SparqlTripleSimple scanTriple{V{"?x"}, V("?y"), V{"?z"}};
     IndexScan scan{qec, Permutation::Enum::PSO, scanTriple};
-    // Note: this currently also contains the (internal) triple for the
-    // `ql:has-pattern` relation of `<x>`.
-    EXPECT_EQ(scan.getSizeEstimate(), 4);
+    EXPECT_EQ(scan.getSizeEstimate(), 3);
   }
   {
     SparqlTripleSimple scanTriple{V{"?x"}, I::fromIriref("<p>"), V{"?y"}};
diff --git a/test/parser/CMakeLists.txt b/test/parser/CMakeLists.txt
index 08b3993d5..ac21f4b45 100644
--- a/test/parser/CMakeLists.txt
+++ b/test/parser/CMakeLists.txt
@@ -1,3 +1,3 @@
 add_subdirectory(data)
 addLinkAndDiscoverTest(ParallelBufferTest parser)
-addLinkAndDiscoverTest(LiteralOrIriTest)
+addLinkAndDiscoverTest(LiteralOrIriTest engine)