adbX/reproscreener_manual_evaluations huggingface dataset:
Manual reproducibility evaluation of arXiv paper 1606.04671
Notes: No code or data released
Code available in article: False
Papers with Code link available: False
Papers with Code link matches: False
Has wrapper scripts: False ()
Software dependencies stated: False ()
Problem stated: False. The contributions of this paper are threefold
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: False. 
Dataset described: False. 
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: True.
Manual reproducibility evaluation of arXiv paper 1903.09668
Notes: No code or data released
Code available in article: False
Papers with Code link available: False
Papers with Code link matches: False
Has wrapper scripts: False ()
Software dependencies stated: False ()
Problem stated: False. 
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: True. Called "Algorithm 1, 2.."
Dataset described: False. Has links to UCI and Kaggle datasets, and a reference to another paper's dataset
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: True. Under the heading "Experiments"
Manual reproducibility evaluation of arXiv paper 1904.10554
Notes: No code or data released
Code available in article: False
Papers with Code link available: False
Papers with Code link matches: False
Has wrapper scripts: False ()
Software dependencies stated: False ()
Problem stated: False. 
Objective stated: False. Rather than taking a model-based approach, it is possible to derive non-parametric reinforcement-learning (RL) methods to study these equilibria. The main idea behind these methods is to directly approximate equilibria from simulations or observed data, providing a powerful alternative to the usual approach
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: True. Called "Algorithm 1, 2.."
Dataset described: False. 
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: True. Under the heading "Experiments"
Manual reproducibility evaluation of arXiv paper 1908.05659
Notes: 90 page review, no experiments, mostly math
Code available in article: False
Papers with Code link available: False
Papers with Code link matches: False
Has wrapper scripts: False ()
Software dependencies stated: False ()
Problem stated: False. 
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: False. 
Dataset described: False. 
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: False.
Manual reproducibility evaluation of arXiv paper 1909.00931
Code available in article: True
Papers with Code link available: True
Papers with Code link matches: True
Papers with Code link notes: Yes, same repo, but still no code (Official Code)
Has wrapper scripts: False ()
Hardware specifications provided: True
Software dependencies stated: False ()
Code language: Python
Problem stated: True. These tasks are central problems in natural language unde
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: True. Called "Algorithm 1, 2.."
Dataset described: True. They link 4 datasets in the paper
Hypothesis stated: True. 
Prediction stated: False
Experiment setup described: True. "Evaluation setting"
Manual reproducibility evaluation of arXiv paper 1911.03867
Notes: No code or data released
Code available in article: False
Papers with Code link available: False
Papers with Code link matches: False
Has wrapper scripts: False ()
Software dependencies stated: False ()
Problem stated: False. 
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: False. 
Dataset described: True. Mentions dataset references and details but not obtainable
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: False.
Manual reproducibility evaluation of arXiv paper 2002.05905
Notes: No code or data shared
Code available in article: False
Papers with Code link available: False
Papers with Code link matches: False
Has wrapper scripts: False ()
Software dependencies stated: False ()
Problem stated: False. 
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: False. 
Dataset described: False. 
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: True. Experimental assessment
Manual reproducibility evaluation of arXiv paper 2004.05258
Code available in article: False
Papers with Code link available: False
Papers with Code link matches: False
Has wrapper scripts: False ()
Software dependencies stated: False ()
Problem stated: False. 
Objective stated: False. 
Research method stated: True. 
Research questions stated: False. 
Pseudocode present: False. 
Dataset described: True. Dataset references
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: True. Setup section under Evaluation Result
Manual reproducibility evaluation of arXiv paper 2009.01947
Notes: Good! Has scripts to reproduce figures. Paper compares algorithms
Code available in article: True
Papers with Code link available: True
Papers with Code link matches: True
Papers with Code link notes: Yes, same repo (Official Code)
Result replication code available: True
Has wrapper scripts: True (gen-table.bash, reproduce.bash)
Software dependencies stated: True (NA/makefile)
Code language: C++, python
Likely to reproduce: True ()
Problem stated: False. 
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: True. Called "Algorithm 1, 2.."
Dataset described: True. We evaluate on a variety of network technologies from the Stanford Large Network Dataset Collection
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: True. Appendix, Additional Experiments
Manual reproducibility evaluation of arXiv paper 2010.04261
Notes: 72 page review
Code available in article: False
Papers with Code link available: False
Papers with Code link matches: False
Has wrapper scripts: False ()
Software dependencies stated: False ()
Problem stated: False. Towards this goal
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: False. 
Dataset described: False. 
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: True. Detailed Experiment Setup
Manual reproducibility evaluation of arXiv paper 2010.04855
Notes: Review, mentions r package for sone of the simulations
Code available in article: False
Papers with Code link available: False
Papers with Code link matches: False
Has wrapper scripts: False ()
Software dependencies stated: False ()
Code language: R?
Likely to reproduce: False ()
Problem stated: False. 
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: False. 
Dataset described: True. We use the dataset published by [26]
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: False.
Manual reproducibility evaluation of arXiv paper 2011.11576
Code available in article: True
Papers with Code link available: True
Papers with Code link matches: False
Papers with Code link notes: Yes, but original repo, not the fork (Official Code)
Is a package: True
Has wrapper scripts: True (sage package, has instructions on load (says in readme))
Hardware specifications provided: True
Software dependencies stated: False (Not sure)
Code language: Sage
Likely to reproduce: False (Too many results and not enough code, somewhat subjective)
Problem stated: True. 
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: True. Called "Algorithm 1, 2.."
Dataset described: True. Dataset references
Hypothesis stated: True. Hypotheses
Prediction stated: False
Experiment setup described: True. Experimental setup
Manual reproducibility evaluation of arXiv paper 2012.09302
Notes: Good! Python package. Well documented and seems reproducible
Code available in article: True
Papers with Code link available: True
Papers with Code link matches: True
Papers with Code link notes: Yes, same repo (Official Code)
Result replication code available: True
Is a package: True
Has wrapper scripts: True (setup.py)
Software dependencies stated: True (requirements.txt)
Likely to reproduce: True ()
Problem stated: False. 
Objective stated: False. 
Research method stated: False. 
Research questions stated: True. RQ_1
Pseudocode present: False. 
Dataset described: True. Details in experimental setup. CIFAR,  ImageNet..
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: True. Experimental Setting
Manual reproducibility evaluation of arXiv paper 2101.07354
Notes: Implements and compares algs from 2 papers, no code in the paper itself
Code available in article: False
Papers with Code link available: False
Papers with Code link matches: False
Has wrapper scripts: False ()
Software dependencies stated: False ()
Problem stated: False. 
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: False. 
Dataset described: True. We refer interested readers to the references above for more detailed information about the three realworld network datasets.
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: True. APPLICATIONS TO REAL-WORLD NETWORKS
Manual reproducibility evaluation of arXiv paper 2102.11887
Notes: Math
Code available in article: False
Papers with Code link available: False
Papers with Code link matches: False
Has wrapper scripts: False ()
Software dependencies stated: False ()
Problem stated: False. 
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: True. 
Dataset described: False. 
Hypothesis stated: True. Proposition 1, 2
Prediction stated: False
Experiment setup described: False.
Manual reproducibility evaluation of arXiv paper 2104.11893
Notes: Seems runnable but not well documented
Code available in article: True
Papers with Code link available: True
Papers with Code link matches: True
Papers with Code link notes: Yes, same repo (Community Code)
Result replication code available: True
Has wrapper scripts: False ()
Software dependencies stated: False ()
Code language: Python
Problem stated: False. 
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: True. Called "Algorithm 1, 2.."
Dataset described: True. Datasets Cora, Citeseer, and Pubmed are three citation benchmark networks widely used in
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: True. Experimental Setting
Manual reproducibility evaluation of arXiv paper 2104.12546
Notes: Some model params described but no code
Code available in article: False
Papers with Code link available: False
Papers with Code link matches: False
Has wrapper scripts: False ()
Software dependencies stated: False ()
Problem stated: True. The problem addressed in this paper is a supervised learning problem.
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: False. 
Dataset described: True. Some datasets cited in refs
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: True. Empirical Study
Manual reproducibility evaluation of arXiv paper 2105.01099
Notes: Review article, no code
Code available in article: False
Papers with Code link available: False
Papers with Code link matches: False
Has wrapper scripts: False ()
Software dependencies stated: False ()
Problem stated: True. We describe the research problems associated
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: False. 
Dataset described: False. 
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: False.
Manual reproducibility evaluation of arXiv paper 2105.01937
Notes: Good! Has conda, and commands for models in paper
Code available in article: True
Papers with Code link available: True
Papers with Code link matches: True
Papers with Code link notes: Yes, same repo (Official Code)
Result replication code available: True
Is a package: False
Has wrapper scripts: True (train.py, evaluate_multiview.py)
Software dependencies stated: True (requirements.txt, 
environment.yml)
Code language: python
Likely to reproduce: True ()
Problem stated: False. 
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: False. 
Dataset described: True. In supplementary material section - More information and access to raw data are provided on the dataset webpage: https://vision.imar.ro/human3.6m/
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: True. Experiments and evaluation
Manual reproducibility evaluation of arXiv paper 2105.15197
Notes: Math and proofs
Code available in article: False
Papers with Code link available: False
Papers with Code link matches: False
Has wrapper scripts: False ()
Software dependencies stated: False ()
Problem stated: False. 
Objective stated: True. The goal of this paper is to
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: False. 
Dataset described: False. 
Hypothesis stated: True. Assumption 3.1
Prediction stated: False
Experiment setup described: False.
Manual reproducibility evaluation of arXiv paper 2106.01528
Notes: Has many other github links to refs
Code available in article: True
Papers with Code link available: True
Papers with Code link matches: True
Papers with Code link notes: Yes, same repo (Official Code)
Result replication code available: True
Has wrapper scripts: True (main.py)
Software dependencies stated: True (pyproject.toml)
Code language: python
Likely to reproduce: True (Partially)
Problem stated: False. 
Objective stated: False. 
Research method stated: True. Methodology section
Research questions stated: False. 
Pseudocode present: True. Called "Algorithm 1, 2.."
Dataset described: True. Feature datasets section
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: True. Experiments section, experimental setting
Manual reproducibility evaluation of arXiv paper 2106.03157
Notes: Good! Replicate.com link. Colab & binder. Has other github links
Code available in article: True
Papers with Code link available: True
Papers with Code link matches: True
Papers with Code link notes: Yes, same repo (Official Code)
Result replication code available: True
Has wrapper scripts: False (Each in its own notebook, readme has instructions)
Software dependencies stated: True (requirements.txt)
Code language: python
Likely to reproduce: True ()
Problem stated: True. Problem is to solve a Rubiks cube - in  abstract but vague in terms of keyword matching
Objective stated: True. Related Work section
Research method stated: True. Proposed Method section
Research questions stated: False. 
Pseudocode present: True. Called "Algorithm 1, 2.."
Dataset described: True. We utilize the publicly available DeepCubeA dataset on Code Ocean2,
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: True. Experiment section
Manual reproducibility evaluation of arXiv paper 2106.03725
Notes: No code or data released
Code available in article: False
Papers with Code link available: False
Papers with Code link matches: False
Has wrapper scripts: False ()
Software dependencies stated: False ()
Problem stated: False. 
Objective stated: True. In this work our goal is to identify the CAD
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: False. 
Dataset described: True. We evaluate our MNN stability results on the Model-Net10 [41] classification problem.
Hypothesis stated: True. Proposition 1, 2
Prediction stated: False
Experiment setup described: True. Numerical Experiments section
Manual reproducibility evaluation of arXiv paper 2106.06927
Code available in article: True
Papers with Code link available: True
Papers with Code link matches: True
Papers with Code link notes: Yes, same repo (Official Code)
Result replication code available: True
Has wrapper scripts: True (scripts/)
Software dependencies stated: True (conda_reqs.txt, pip_reqs.txt)
Code language: python
Likely to reproduce: True (Partially)
Problem stated: False. 
Objective stated: False. 
Research method stated: True. Proposed Method section
Research questions stated: False. 
Pseudocode present: False. 
Dataset described: True. Standard datasets - CIFAR, ImageNet
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: True. Multiple Experimental Setup sections for each technique
Manual reproducibility evaluation of arXiv paper 2106.07704
Notes: They have instructions for each experiment run
Code available in article: True
Papers with Code link available: True
Papers with Code link matches: True
Papers with Code link notes: Yes, same repo (Official Code)
Result replication code available: True
Has wrapper scripts: True (Dockerfile, run_experiments.py)
Software dependencies stated: True (requirements.txt)
Code language: python
Likely to reproduce: True ()
Problem stated: True. 
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: True. Called "Algorithm 1, 2.."
Dataset described: True. We sub-sampled 50k training examples from the SNLI dataset (Bowman et al., 2015),
Hypothesis stated: False. Note that hypothesis and hypotheses are words that are part of the work: (premise, hypothesis) sentence pairs,
Prediction stated: False
Experiment setup described: True. Applications and Experiments sections
Manual reproducibility evaluation of arXiv paper 2106.10898
Notes: No code released
Code available in article: False
Papers with Code link available: False
Papers with Code link matches: False
Has wrapper scripts: False ()
Software dependencies stated: False ()
Problem stated: True. To address the aforementioned problems,
Objective stated: False. 
Research method stated: True. Methodologies section/chapter
Research questions stated: False. 
Pseudocode present: False. 
Dataset described: True. Dataset. For this experiment, the personalization dataset3 has
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: True. Experiments and Implementation section
Manual reproducibility evaluation of arXiv paper 2106.12177
Notes: Theory paper
Code available in article: False
Papers with Code link available: False
Papers with Code link matches: False
Has wrapper scripts: False ()
Software dependencies stated: False ()
Problem stated: False. 
Objective stated: False. 
Research method stated: True. MAIN RESEARCH TOPICS AND METHODS section
Research questions stated: False. 
Pseudocode present: True. Called "Algorithm 1, 2.."
Dataset described: False. 
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: False.
Manual reproducibility evaluation of arXiv paper 2106.12936
Notes: Math and proofs
Code available in article: False
Papers with Code link available: False
Papers with Code link matches: False
Has wrapper scripts: False ()
Software dependencies stated: False ()
Problem stated: False. 
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: False. 
Dataset described: False. 
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: False.
Manual reproducibility evaluation of arXiv paper 2106.13823
Notes: Theory paper
Code available in article: False
Papers with Code link available: False
Papers with Code link matches: False
Has wrapper scripts: False ()
Software dependencies stated: False ()
Problem stated: False. 
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: False. 
Dataset described: False. 
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: False.
Manual reproducibility evaluation of arXiv paper 2107.01131
Notes: Code inside a zip file in repo. Most sections are "to be updated"
Code available in article: True
Papers with Code link available: True
Papers with Code link matches: True
Papers with Code link notes: Yes, same repo (Official Code)
Result replication code available: True
Has wrapper scripts: False (Incomplete repo (says to be updated))
Software dependencies stated: False (Incomplete repo (says to be updated))
Code language: python
Likely to reproduce: False ()
Problem stated: False. 
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: True. Called "Algorithm 1, 2.."
Dataset described: False. 
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: True. Experiments section
Manual reproducibility evaluation of arXiv paper 2108.09779
Notes: Their link in the abstract is a github.io link (actual github link not in paper). Papers with code does not point to the right one (points to their reference/previous work).
Code available in article: True
Papers with Code link available: True
Papers with Code link matches: False
Papers with Code link notes: Yes, different repo/previous work (Official Code) https://github.com/Denys88/rl_games
Result replication code available: False
Is a package: True
Has wrapper scripts: True (scripts/rlg_hydra.p)
Software dependencies stated: True (setup.py (package))
Code language: python
Likely to reproduce: True (Partially)
Problem stated: True. 
Objective stated: False. 
Research method stated: True. Methods section
Research questions stated: True. In our experiments, we aim to answer the following four questions
Pseudocode present: False. 
Dataset described: True. EGAD dataset
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: True. Experiments section
Manual reproducibility evaluation of arXiv paper 2109.01372
Notes: Good! Scripts to reproduce plots and experiments
Code available in article: True
Papers with Code link available: True
Papers with Code link matches: True
Papers with Code link notes: Yes, same repo. (Two links but both redirec to the same directory. (Official Code)
Result replication code available: True
Has wrapper scripts: True (plots.py, run.py)
Software dependencies stated: True (pip install cardinal in readme)
Likely to reproduce: True ()
Problem stated: False. 
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: True. Called "Algorithm 1, 2.."
Dataset described: True. Standard datasets - CIFAR, MNIST, Fashion
Hypothesis stated: True. We hypothesize that this side-effect of diversity contributes to its success
Prediction stated: False
Experiment setup described: True. Experiments section
Manual reproducibility evaluation of arXiv paper 2109.12784
Notes: Good! Script to run all experiments
Code available in article: True
Papers with Code link available: True
Papers with Code link matches: True
Papers with Code link notes: Yes, same repo (Official Code)
Result replication code available: True
Has wrapper scripts: True (main.py)
Software dependencies stated: True (Typed out in readme)
Code language: python
Likely to reproduce: True ()
Problem stated: True. Motivated by the problem of learning with
Objective stated: False. 
Research method stated: False. 
Research questions stated: False. 
Pseudocode present: False. 
Dataset described: True. Standard datasets - CIFAR, MNIST, Fashion
Hypothesis stated: False. 
Prediction stated: False
Experiment setup described: False. Details of Experimental evaluation section