opened
aveiros
username_0
## My Environment

* __ArangoDB Version__:        3.4.6
* __Storage Engine__:             RocksDB
* __Deployment Mode__:       Single Server
* __Deployment Strategy__:    ArangoDB Starter in Docker
* __Infrastructure__:               own
* __Operating System__:        MacOS 10.13.4
* __Total RAM in your machine__:        32Gb
* __Disks in use__:        SSD
* __Used Package__:              Docker - official Docker library

While using the `/_api/import` endpoint and writing the **same data** _concurrently_ to a collection, views liked to that collection eventually lose records.

Steps to reproduce

1) Spin up Arango
```
docker run -dt --name arangodb346-bug -e ARANGO_NO_AUTH=1 -p 8529:8529 arangodb/arangodb:3.4.6
```
2) Create a Collection named `docs` on _system db
3) Create a View named `docs_view` on _system db and link to the `docs` Collection (setup i used below)
```
"links": {
  "docs": {
    "analyzers": [
      "identity"
    ],
    "fields": {},
    "includeAllFields": true,
    "storeValues": "id",
    "trackListPositions": false
  }
}
```
4) Wrote a javascript to reproduce the issue (requires node)
Create a `package.json` file
```
{
    "dependencies": {
        "async": "^3.1.0",
        "request": "^2.88.0",
        "uuid": "^3.3.2"
    }
}
```

Create a `script.js` file
```
const request = require("request");
const uuidv1 = require("uuid/v1");
const async = require("async");

let docs = [];
for (let i = 0; i < 100; i++) {
    let docId = uuidv1();
    docs.push({ _id: "docs/" + docId, _key: docId });
}

let requests = [];
for (let j = 0; j < 1000; j++) {
    let requestFunc = function(cb) {
        let options = {
            method: "POST",
            url: "http://localhost:8529/_db/_system/_api/import?collection=docs&type=list&details=true",
            // url: "http://localhost:8529/_db/_system/_api/import?collection=docs&type=list&details=true&onDuplicate=replace",
            body: docs,
            json: true
        };
    
        request(options, function (err, res, body) {
            console.log(j, new Date(), 'response code:', res.statusCode, 'created:', body.created, 'errors:', body.errors, 'empty:', body.empty, 'ignored:', body.ignored, 'updated:', body.updated);
            cb();
        });
    };

    requests.push(requestFunc);
}

const args = process.argv.slice(2);
if(args.length) {
    if("sequential" == args[0]) {
        async.waterfall(requests, function() {
            console.log("sequential done");
        });
    }

    if("parallel" == args[0]) {
        async.parallelLimit(requests, 10, function() {
            console.log("parallel done");
        });
    }
}
```

Perform a `npm install`

5) Execute the following command (the below)
```node script.js parallel```

6) Perform following query
```
LET collectionCount = FIRST(FOR u IN docs COLLECT WITH COUNT INTO length RETURN length)
LET viewCount = LENGTH(FOR d IN docs_view RETURN d)
RETURN { collectionCount, viewCount, diff: (collectionCount - viewCount) }
```

Eventually the view gets out of sync with the collection. Repeat 5 and 6 if necessary.

Few notes
1) Dropping `onDuplicate=replace` from the request, becomes less frequent, but still happens (takes longer)
2) Using ```node script.js sequential```, becomes less frequent, but still happens (takes longer)
Concurrent Writes leads to data loss in ArangoSearch/Views
issue
created
aveiros
username_0
Reproduced on the `/_api/document/{collection}` endpoint.

1) Following the same setup for Collection/View Above.
2) Create the following scripts as `script-single.js`
```
const request = require("request");
const uuidv1 = require("uuid/v1");
const async = require("async");

let docId = uuidv1();
let doc = { _id: "docs/" + docId, _key: docId };

let requests = [];
for (let j = 0; j < 1000; j++) {
    let requestFunc = function(cb) {
        let options = {
            method: "POST",
            url: "http://localhost:8529/_db/_system/_api/document/docs?overwrite=true",
            body: doc,
            json: true
        };

        request(options, function (err, res, body) {
            console.log(j, new Date(), 'response code:', res.statusCode);
            cb();
        });
    };

    requests.push(requestFunc);
}

const args = process.argv.slice(2);
if(args.length) {
    if("sequential" == args[0]) {
        async.waterfall(requests, function() {
            console.log("sequential done");
        });
    }

    if("parallel" == args[0]) {
        async.parallelLimit(requests, 10, function() {
            console.log("parallel done");
        });
    }
}
```
3) Execute the following command

```for i in {1..5}; do (node script-script.js parallel &); done``` // to hammer it
or 
```node script-script.js parallel``` // less visible

6) Perform the following query
```
LET collectionCount = FIRST(FOR u IN docs COLLECT WITH COUNT INTO length RETURN length)
LET viewCount = LENGTH(FOR d IN docs_view RETURN d)
RETURN { collectionCount, viewCount, diff: (collectionCount - viewCount) }
```
comment
created
maxkernbach
username_1
Hi @username_0,

Thanks for reporting this issue.
We have been able to reproduce the bug and are currently working on a fix.
comment
closed
maxkernbach
username_1
issue
created
maxkernbach
username_1
Hi @username_0 and @davechallis,

Both versions 3.4.8 and 3.5.1, which include a fix for the data loss, have been released. In case you are still observing data loss in ArangoSearch Views, please comment and we will reopen this issue.

You can find the latest release in the [download section](https://www.arangodb.com/download-major/).
comment