Skill: killgfat/pubmed-edirect (version 0.4.4)

--- name: pubmed-edirect description: Search and retrieve literature from PubMed using NCBI's EDirect command-line tools. ⚠️ Advanced skill requiring manual installation. requires: bins: - esearch - efetch - elink - xtract - einfo - efilter install: - id: edirect kind: manual label: Manual Installation Required - Review INSTALL.md docs: https://www.ncbi.nlm.nih.gov/books/NBK179288/ note: "⚠️ User must manually download and review official installer script" security_level: elevated metadata: openclaw: emoji: 🔬 category: advanced security_level: elevated requires: bins: - esearch - efetch - elink - xtract - einfo - efilter env: - name: NCBI_API_KEY optional: true description: NCBI API key for increased rate limits (10 requests/sec vs 3 requests/sec) - name: NCBI_EMAIL optional: true description: Email address to identify yourself to NCBI --- # PubMed EDirect Skill Search and retrieve literature from PubMed using NCBI's EDirect command-line tools. ## ⚠️ Security Advisory **Important**: This skill requires installation of external command-line tools. The installation process involves: 1. **External script execution**: Downloading and executing installation scripts from the official NCBI FTP server 2. **System modifications**: Adding directories to your PATH environment variable 3. **Permission requirements**: May require installation of Perl modules and dependencies **Before installation, you must**: 1. Review the installer script content after downloading 2. Confirm the source is trustworthy (official `ftp.ncbi.nlm.nih.gov` domain) 3. Validate in a test environment 4. Understand all commands that will be executed ## Overview This skill provides access to PubMed and other NCBI databases through the official EDirect (Entrez Direct) utilities. EDirect is a suite of programs that provide access to the NCBI's suite of interconnected databases (publication, sequence, structure, gene, variation, expression, etc.) from Unix terminals. **Note: This is a local installation skill** – all tools run directly on your system without Docker or containerization. Follow the [INSTALL.md](INSTALL.md) guide for local setup. ## Structure The skill is organized into the following files: - **`INSTALL.md`** - Installation and configuration guide - **`BASICS.md`** - Basic usage and common commands - **`ADVANCED.md`** - Advanced techniques and complex queries - **`EXAMPLES.md`** - Practical usage examples - **`REFERENCE.md`** - Quick reference (field qualifiers, formats, etc.) - **`OPENCLAW_INTEGRATION.md`** - OpenClaw-specific usage guide - **`scripts/`** - Useful bash scripts for common tasks ## Quick Start 1. **Read the installation guide**: Review [INSTALL.md](INSTALL.md) for secure installation steps 2. **Manually install EDirect**: ```bash # Step 1: Download the script wget -q https://ftp.ncbi.nlm.nih.gov/entrez/entrezdirect/install-edirect.sh # Step 2: Review content (important for security) less install-edirect.sh # Step 3: Execute installation ./install-edirect.sh ``` 3. **Verify installation**: ```bash esearch -db pubmed -query "test" -retmax 1 ``` 4. **Explore examples**: Check [EXAMPLES.md](EXAMPLES.md) ## Core Tools The skill provides access to EDirect tools through OpenClaw's `exec` capability: - `esearch` - Search databases - `efetch` - Retrieve records - `elink` - Find related records - `efilter` - Filter results - `xtract` - Extract data from XML - `einfo` - Get database information ## Databases Supported EDirect supports numerous NCBI databases including: - `pubmed` - Biomedical literature - `pmc` - PubMed Central full-text articles - `gene` - Gene information - `nuccore` - Nucleotide sequences - `protein` - Protein sequences - `mesh` - Medical Subject Headings - And many more... ## Key Features - **Command-line access** to NCBI databases - **Pipeline architecture** using Unix pipes - **Structured data extraction** with XML parsing - **Batch processing** capabilities - **Cross-database linking** between records ## Getting Help - Use `-help` with any EDirect command: `esearch -help` - Consult the [official documentation](https://www.ncbi.nlm.nih.gov/books/NBK179288/) - Check troubleshooting in installation guide ## Included Scripts The `scripts/` directory contains ready-to-use bash scripts: ### `batch_fetch_abstracts.sh` Fetch abstracts for a list of PMIDs with error handling and rate limiting. ```bash ./scripts/batch_fetch_abstracts.sh pmids.txt abstracts/ 0.5 ``` ### `search_export_csv.sh` Search PubMed and export results to CSV with metadata. ```bash ./scripts/search_export_csv.sh "CRISPR [TIAB]" 100 results.csv ``` ### `publication_trends.sh` Analyze publication trends over time with visualization. ```bash ./scripts/publication_trends.sh "machine learning" 2010 2023 trends.csv ``` ## Security Best Practices ### 1. Script Review ```bash # Always download first and review scripts wget -q SOURCE_URL -O script.sh less script.sh # or cat script.sh | head -50 # Execute only after review ./script.sh ``` ### 2. Environment Isolation - Running in Docker containers provides isolation - Use virtual machines for testing - Set up dedicated user accounts ### 3. Least Privilege - Do not run as root - Set appropriate file permissions - Use dedicated directories for data ### 4. Network Controls - Configure firewall rules - Use proxies for controlled access - Monitor network traffic ## Notes **Important**: This skill requires manual installation and configuration. All installation steps require explicit user confirmation and execution. This skill provides command-line access to NCBI databases through local installation of EDirect tools.

path: ADVANCED.md
content: # Advanced EDirect Techniques

## Complex XML Extraction with xtract

### Nested Exploration

```bash
# Extract authors with their affiliations
efetch -db pubmed -id 1234567 -format xml | \
  xtract -pattern PubmedArticle \
    -block Author \
      -element LastName,Initials \
      -block AffiliationInfo -element Affiliation
```

### Conditional Extraction

```bash
# Extract only corresponding authors
efetch -db pubmed -id 1234567 -format xml | \
  xtract -pattern PubmedArticle \
    -block Author \
    -if "@ValidYN" -equals "Y" \
    -element LastName,Initials
```

### Variable Usage in xtract

```bash
# Store and use variables
efetch -db pubmed -id 1234567 -format xml | \
  xtract -pattern PubmedArticle \
    -YEAR Year \
    -block Author \
    -element "&YEAR" LastName,Initials
```

## Multi-Step Query Pipelines

### Complex Cross-Database Queries

```bash
# Find genes associated with diseases in literature
esearch -db pubmed -query "alzheimer disease" | \
  elink -target gene | \
  efilter -organism human | \
  efetch -format docsum | \
  xtract -pattern DocumentSummary -element Id Name Description
```

### Chaining Multiple Filters

```bash
# Highly specific search
esearch -db pubmed -query "cancer" | \
  efilter -mindate 2020 -maxdate 2023 | \
  efilter -query "review [PTYP]" | \
  efilter -query "english [LANG]" | \
  efilter -query "humans [MESH]" | \
  efetch -format abstract
```

## Advanced Search Techniques

### Using Search History

```bash
# Save search to history server
esearch -db pubmed -query "complex query" -log > search_history.log

# Use WebEnv from history
esearch -db pubmed -query "additional terms" -webenv "$WEBENV" -query_key "$QUERY_KEY"
```

### Combining Multiple Searches

```bash
# Intersection of two searches
search1=$(esearch -db pubmed -query "term1" | efetch -format uid)
search2=$(esearch -db pubmed -query "term2" | efetch -format uid)

# Find common PMIDs
comm -12 <(echo "$search1" | sort) <(echo "$search2" | sort) | \
  efetch -db pubmed -format abstract
```

### Proximity Searching

```bash
# Words near each other (using PubMed syntax)
esearch -db pubmed -query '"machine learning"~10' | efetch -format abstract
```

## Data Transformation Pipelines

### Convert XML to Structured Formats

```bash
# Create CSV with multiple fields
esearch -db pubmed -query "genomics" -retmax 100 | \
  efetch -format xml | \
  xtract -pattern PubmedArticle \
    -element MedlineCitation/PMID \
    -block PubDate -sep "-" -element Year,Month,Day \
    -block Author -position first -sep " " -element LastName,Initials \
    -element Article/ArticleTitle \
    -block Abstract -element AbstractText | \
  awk 'BEGIN {FS="\t"; OFS=","} {print $1,$2,$3,$4,$5,$6}' > output.csv
```

### Extract and Analyze Citation Networks

```bash
# Build citation graph for a paper
elink -db pubmed -id 1234567 -cited | \
  efetch -format uid | \
  while read cited_pmid; do
    echo "1234567 -> $cited_pmid"
    # Get papers citing the cited paper
    elink -db pubmed -id "$cited_pmid" -cited | \
      efetch -format uid | \
      while read cited2; do
        echo "$cited_pmid -> $cited2"
      done
    sleep 0.5
  done > citation_network.tsv
```

## Performance Optimization

### Parallel Processing

```bash
# Process multiple PMIDs in parallel
cat pmids.txt | \
  xargs -P 4 -I {} bash -c '
    efetch -db pubmed -id {} -format abstract > abstract_{}.txt
    sleep 0.3
  '
```

### Caching Strategies

```bash
# Cache XML for repeated queries
if [ ! -f "cached_query.xml" ]; then
  esearch -db pubmed -query "expensive query" | \
    efetch -format xml > cached_query.xml
fi

# Use cached data
cat cached_query.xml | \
  xtract -pattern PubmedArticle -element Title
```

### Batch Processing with Error Recovery

```bash
# Process with retry logic
process_pmid() {
  local pmid=$1
  local attempt=1
  local max_attempts=3
  
  while [ $attempt -le $max_attempts ]; do
    if efetch -db pubmed -id "$pmid" -format abstract > "abstract_${pmid}.txt" 2>/dev/null; then
      echo "Processed $pmid"
      return 0
    else
      echo "Attempt $attempt failed for $pmid"
      sleep $((attempt * 2))
      attempt=$((attempt + 1))
    fi
  done
  echo "Failed to process $pmid after $max_attempts attempts" >&2
  return 1
}

export -f process_pmid
cat pmids.txt | xargs -P 2 -I {} bash -c 'process_pmid "$@"' _ {}
```

## Advanced xtract Features

### Pattern Matching with Regular Expressions

```bash
# Extract emails from affiliations
efetch -db pubmed -id 1234567 -format xml | \
  xtract -pattern PubmedArticle \
    -block AffiliationInfo \
    -if Affiliation -matches "@" \
    -element Affiliation
```

### Statistical Operations

```bash
# Calculate average authors per paper
esearch -db pubmed -query "your topic" -retmax 50 | \
  efetch -format xml | \
  xtract -pattern PubmedArticle \
    -NUM Author \
    -block Author -last "" | \
  awk '{sum += $1; count++} END {print "Average authors:", sum/count}'
```

### Hierarchical Data Extraction

```bash
# Extract MeSH terms with qualifiers
efetch -db pubmed -id 1234567 -format xml | \
  xtract -pattern PubmedArticle \
    -block MeshHeading \
      -element DescriptorName \
      -subset QualifierName -element QualifierName
```

## Database-Specific Advanced Queries

### Gene Expression Data

```bash
# Link genes to GEO datasets
esearch -db gene -query "TP53 AND human" | \
  elink -target gds | \
  efetch -format docsum | \
  xtract -pattern DocumentSummary -element Id Title Summary
```

### Protein Structure Links

```bash
# Find protein structures for a gene
esearch -db gene -query "INSR AND human" | \
  elink -target structure | \
  efetch -format docsum | \
  xtract -pattern DocumentSummary -element Id PdbAcc Title
```

### Clinical Trial Connections

```bash
# Link publications to clinical trials
esearch -db pubmed -query "clinical trial phase III" | \
  elink -target clinvar | \
  efetch -format docsum | \
  xtract -pattern DocumentSummary -element Id ClinicalSignificance Name
```

## Scripting with EDirect

### Create Reusable Functions

```bash
#!/bin/bash
# pmid_to_authors.sh - Extract authors from PMID

PMID_TO_AUTHORS() {
  local pmid=$1
  efetch -db pubmed -id "$pmid" -format xml | \
    xtract -pattern PubmedArticle \
      -block Author \
      -sep ", " -element LastName,Initials
}

# Usage
PMID_TO_AUTHORS 1234567
```

### Build Custom Analysis Pipelines

```bash
#!/bin/bash
# literature_trends.sh - Analyze publication trends

analyze_trends() {
  local query="$1"
  local start_year="$2"
  local end_year="$3"
  
  for year in $(seq "$start_year" "$end_year"); do
    count=$(esearch -db pubmed -query "$query" | \
      efilter -mindate "$year" -maxdate "$year" | \
      xtract -pattern Count -element Count)
    echo "$year,$count"
    sleep 1
  done
}

analyze_trends "artificial intelligence" 2010 2023 > ai_trends.csv
```

## Integration with Other Tools

### Combine with jq for JSON Processing

```bash
# Convert to JSON and process
esearch -db pubmed -query "test" | \
  efetch -format xml | \
  xml2json | \
  jq '.PubmedArticleSet.PubmedArticle[].MedlineCitation.PMID' 
```

### Use with Python/R for Analysis

```bash
# Export data for Python analysis
esearch -db pubmed -query "cancer biomarkers" -retmax 100 | \
  efetch -format xml | \
  xtract -pattern PubmedArticle \
    -element PMID Year Title | \
  awk 'BEGIN {FS="\t"; print "pmid,year,title"} {print $1","$2",\""$3"\""}' > \
  data_for_analysis.csv

# Then in Python:
# import pandas as pd
# df = pd.read_csv('data_for_analysis.csv')
```

### Database Integration

```bash
# Load results into SQLite
esearch -db pubmed -query "your query" -retmax 1000 | \
  efetch -format xml | \
  xtract -pattern PubmedArticle \
    -element PMID Year Title | \
  awk 'BEGIN {FS="\t"; print "BEGIN TRANSACTION;"} 
       {print "INSERT INTO publications VALUES (\"" $1 "\", \"" $2 "\", \"" $3 "\");"} 
       END {print "COMMIT;"}' | \
  sqlite3 literature.db
```

## Monitoring and Debugging

### Query Performance Analysis

```bash
# Time your queries
time esearch -db pubmed -query "complex query" | efetch -format abstract > /dev/null
```

### Debug XML Structure

```bash
# See XML structure of a record
efetch -db pubmed -id 1234567 -format xml | \
  xtract -outline | head -20
```

### Validate Queries

```bash
# Test query without fetching
esearch -db pubmed -query "test query" -retmax 0 | \
  xtract -pattern Count -element Count
```

## Best Practices for Production Use

1. **Always use an API key** for higher rate limits
2. **Implement caching** for repetitive queries
3. **Add error handling** and retry logic
4. **Respect rate limits** with appropriate delays
5. **Log your queries** for reproducibility
6. **Validate results** before processing
7. **Use version control** for analysis scripts
8. **Document your pipelines** for future reference

## Next Steps

- Apply these techniques to your specific research questions
- Combine EDirect with other bioinformatics tools
- Automate repetitive literature review tasks
- Build custom dashboards for monitoring publication trends

path: BASICS.md
content: # Basic EDirect Usage

## Core Command Overview

EDirect consists of several commands that work together through Unix pipes:

| Command | Purpose | Common Use |
|---------|---------|------------|
| `esearch` | Search databases | `esearch -db pubmed -query "cancer"` |
| `efetch` | Retrieve records | `efetch -format abstract` |
| `elink` | Find related records | `elink -related` |
| `efilter` | Filter results | `efilter -mindate 2020` |
| `xtract` | Extract data from XML | `xtract -pattern PubmedArticle -element Title` |
| `einfo` | Database information | `einfo -db pubmed` |

## Basic Search Workflow

### 1. Simple Search and Retrieve

```bash
# Search and get abstracts
esearch -db pubmed -query "alzheimer disease" | efetch -format abstract
```

### 2. Search with Results Limit

```bash
# Get first 10 results
esearch -db pubmed -query "diabetes" -retmax 10 | efetch -format abstract
```

### 3. Search in Specific Field

```bash
# Search in title/abstract only
esearch -db pubmed -query "CRISPR [TIAB]" | efetch -format medline
```

## Fetching Records

### By Search Results

```bash
# Standard pipeline
esearch -db pubmed -query "machine learning" | efetch -format xml
```

### By Direct PMIDs

```bash
# Single PMID
efetch -db pubmed -id 1234567 -format abstract

# Multiple PMIDs
efetch -db pubmed -id 1234567,2345678,3456789 -format medline

# From a file
cat pmids.txt | efetch -db pubmed -format abstract
```

### Output Formats

| Format | Description | Use Case |
|--------|-------------|----------|
| `abstract` | Human-readable abstract | Reading articles |
| `medline` | MEDLINE format | Reference managers |
| `xml` | Structured XML | Data extraction |
| `uid` | PMIDs only | Getting IDs |
| `docsum` | Document summary | Quick overview |

## Linking Between Records

### Find Related Articles

```bash
# Articles related to search results
esearch -db pubmed -query "parkinson disease" | elink -related | efetch -format abstract
```

### Citation Tracking

```bash
# Articles citing a specific paper
elink -db pubmed -id 1234567 -cited | efetch -format abstract

# Articles cited by a paper
elink -db pubmed -id 1234567 -cites | efetch -format abstract
```

### Cross-Database Links

```bash
# Link PubMed articles to PMC (full text)
esearch -db pubmed -query "nature 2023" | elink -target pmc | efetch -format xml
```

## Filtering Results

### By Date

```bash
# Articles from specific period
esearch -db pubmed -query "cancer" | \
  efilter -mindate 2020 -maxdate 2023 | \
  efetch -format abstract
```

### By Publication Type

```bash
# Review articles only
esearch -db pubmed -query "immunotherapy" | \
  efilter -query "review [PTYP]" | \
  efetch -format abstract
```

### By Language

```bash
# English articles only
esearch -db pubmed -query "genomics" | \
  efilter -query "english [LANG]" | \
  efetch -format abstract
```

## Basic Data Extraction with xtract

### Extract Simple Fields

```bash
# Get titles and PMIDs
esearch -db pubmed -query "COVID-19" | \
  efetch -format xml | \
  xtract -pattern PubmedArticle -element MedlineCitation/PMID Article/ArticleTitle
```

### Extract Authors

```bash
# Get author lists
efetch -db pubmed -id 1234567 -format xml | \
  xtract -pattern PubmedArticle -block Author -sep " " -element LastName,Initials
```

### Count Results

```bash
# Count search results
esearch -db pubmed -query "vaccine" | \
  xtract -pattern Count -element Count
```

## Working with Different Databases

### PubMed (Literature)

```bash
# Standard PubMed search
esearch -db pubmed -query "your query" | efetch -format abstract
```

### PubMed Central (Full Text)

```bash
# Search full-text articles
esearch -db pmc -query "open access" | efetch -format xml
```

### Gene Database

```bash
# Search for genes
esearch -db gene -query "BRCA1 AND human" | efetch -format docsum
```

### Nucleotide Sequences

```bash
# Search nucleotide database
esearch -db nuccore -query "insulin [PROT]" | efetch -format fasta
```

## Common Query Patterns

### Boolean Operators

```bash
# AND operator
esearch -db pubmed -query "cancer AND therapy"

# OR operator  
esearch -db pubmed -query "(lung OR breast) AND cancer"

# NOT operator
esearch -db pubmed -query "cancer NOT review"
```

### Phrase Searching

```bash
# Exact phrase
esearch -db pubmed -query '"clinical trial"'

# With field qualifier
esearch -db pubmed -query '"randomized controlled trial" [TIAB]'
```

### Truncation/Wildcards

```bash
# Word stem searching
esearch -db pubmed -query "immun*"  # Finds immune, immunity, immunization, etc.
```

## Managing Output

### Save to Files

```bash
# Save abstracts to file
esearch -db pubmed -query "neuroscience" | \
  efetch -format abstract > abstracts.txt

# Save PMIDs to file
esearch -db pubmed -query "genetics" | \
  efetch -format uid > pmids.txt
```

### Format Output

```bash
# Pretty print XML
esearch -db pubmed -query "test" | \
  efetch -format xml | xmllint --format -

# Convert to CSV
esearch -db pubmed -query "test" | \
  efetch -format xml | \
  xtract -pattern PubmedArticle -tab "," -element PMID,Title | \
  sort -t, -k2 > output.csv
```

## Error Handling

### Check for Errors

```bash
# Add -log option to see details
esearch -db pubmed -query "test" -log | efetch -format abstract
```

### Handle Empty Results

```bash
# Check count first
count=$(esearch -db pubmed -query "very specific term" | \
  xtract -pattern Count -element Count)
if [ "$count" -gt 0 ]; then
  # Process results
  esearch -db pubmed -query "very specific term" | efetch -format abstract
else
  echo "No results found"
fi
```

## Performance Tips

### Use Appropriate Limits

```bash
# Don't fetch more than needed
esearch -db pubmed -query "broad topic" -retmax 50 | efetch -format abstract
```

### Add Delays for Batch Processing

```bash
# Process list with delay
while read pmid; do
  efetch -db pubmed -id "$pmid" -format abstract
  sleep 0.5  # Respect rate limits
done < pmids.txt
```

### Cache Results

```bash
# Save intermediate results
esearch -db pubmed -query "complex query" -retmax 100 > search_results.xml
cat search_results.xml | efetch -format abstract
```

## Next Steps

- See [EXAMPLES.md](EXAMPLES.md) for practical use cases
- Explore [ADVANCED.md](ADVANCED.md) for complex queries
- Refer to [REFERENCE.md](REFERENCE.md) for field qualifiers and formats

path: EXAMPLES.md
content: # EDirect Practical Examples

## Literature Review Examples

### 1. Basic Literature Search

```bash
# Search for articles on a topic and get abstracts
esearch -db pubmed -query "CRISPR gene editing" | \
  efetch -format abstract | head -500
```

### 2. Recent Review Articles

```bash
# Find recent review articles on a topic
esearch -db pubmed -query "cancer immunotherapy" | \
  efilter -mindate 2022 -query "review [PTYP]" | \
  efetch -format abstract | head -200
```

### 3. Author Publication List

```bash
# Get publications by a specific author
esearch -db pubmed -query "Zhang Y [AUTH]" | \
  efetch -format medline | grep -A2 -B2 "TI\|AB" | head -100
```

## Data Extraction Examples

### 4. Extract Titles and PMIDs to CSV

```bash
# Create CSV of search results
esearch -db pubmed -query "machine learning healthcare" -retmax 50 | \
  efetch -format xml | \
  xtract -pattern PubmedArticle \
    -tab "," -element MedlineCitation/PMID Article/ArticleTitle | \
  awk 'BEGIN {print "pmid,title"} {print}' > ml_healthcare.csv
```

### 5. Get Author Information

```bash
# Extract all authors from specific papers
for pmid in 1234567 2345678 3456789; do
  echo "=== PMID: $pmid ==="
  efetch -db pubmed -id "$pmid" -format xml | \
    xtract -pattern PubmedArticle \
      -block Author \
      -sep ", " -element LastName,Initials
  echo
done
```

### 6. Extract Publication Dates

```bash
# Analyze publication timeline
esearch -db pubmed -query "COVID-19 vaccine" -retmax 100 | \
  efetch -format xml | \
  xtract -pattern PubmedArticle \
    -block PubDate \
    -sep "-" -element Year,Month | \
  sort | uniq -c | sort -rn
```

## Cross-Database Examples

### 7. Link Papers to Genes

```bash
# Find genes mentioned in Alzheimer's research
esearch -db pubmed -query "Alzheimer disease" -retmax 50 | \
  elink -target gene | \
  efilter -organism human | \
  efetch -format docsum | \
  xtract -pattern DocumentSummary -element Id Name Description | head -20
```

### 8. Find Full-Text Articles

```bash
# Get PMC IDs for open access articles
esearch -db pubmed -query "nature 2023 open access" | \
  elink -target pmc | \
  efetch -format uid | \
  while read pmcid; do
    echo "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC${pmcid}/"
  done
```

### 9. Protein Sequences from Literature

```bash
# Find protein sequences studied in papers
esearch -db pubmed -query "insulin receptor" | \
  elink -target protein | \
  efilter -organism human | \
  efetch -format fasta | head -1000
```

## Analysis Examples

### 10. Publication Trend Analysis

```bash
# Count publications per year
for year in {2010..2023}; do
  count=$(esearch -db pubmed -query "artificial intelligence" | \
    efilter -mindate "$year" -maxdate "$year" | \
    xtract -pattern Count -element Count)
  echo "$year: $count publications"
  sleep 1
done
```

### 11. Journal Analysis

```bash
# Most common journals in search results
esearch -db pubmed -query "genomics" -retmax 500 | \
  efetch -format xml | \
  xtract -pattern PubmedArticle \
    -element Journal/Title | \
  sort | uniq -c | sort -rn | head -10
```

### 12. Collaboration Network

```bash
# Extract authors and find collaborations
esearch -db pubmed -query "single cell RNA sequencing" -retmax 100 | \
  efetch -format xml | \
  xtract -pattern PubmedArticle \
    -block Author \
    -sep " " -element LastName,Initials | \
  sort | uniq -c | sort -rn | head -20
```

## Batch Processing Examples

### 13. Download Abstracts for PMID List

```bash
# Process file of PMIDs
cat pmids.txt | \
  while read pmid; do
    echo "=== PMID: $pmid ===" >> abstracts.txt
    efetch -db pubmed -id "$pmid" -format abstract >> abstracts.txt
    echo "" >> abstracts.txt
    sleep 0.5
  done
```

### 14. Bulk Export to RIS Format

```bash
# Export for reference manager
esearch -db pubmed -query "your search" -retmax 100 | \
  efetch -format medline > references.ris
```

### 15. Parallel Processing of Queries

```bash
# Search multiple terms in parallel
search_terms=("cancer" "diabetes" "alzheimer" "cardiovascular")
for term in "${search_terms[@]}"; do
  (
    esearch -db pubmed -query "$term" -retmax 50 | \
      efetch -format xml > "${term}_results.xml"
    echo "Completed: $term"
  ) &
done
wait
```

## Real-World Research Scenarios

### 16. Systematic Review Helper

```bash
#!/bin/bash
# systematic_review.sh - Aid in systematic review

QUERY="$1"
OUTPUT_DIR="systematic_review_$(date +%Y%m%d)"
mkdir -p "$OUTPUT_DIR"

# Get total count
total=$(esearch -db pubmed -query "$QUERY" | \
  xtract -pattern Count -element Count)
echo "Total results: $total"

# Fetch in batches
for offset in $(seq 0 100 $total); do
  esearch -db pubmed -query "$QUERY" -retstart $offset -retmax 100 | \
    efetch -format xml > "$OUTPUT_DIR/batch_${offset}.xml"
  echo "Fetched batch starting at $offset"
  sleep 2
done

# Extract key information
cat "$OUTPUT_DIR"/*.xml | \
  xtract -pattern PubmedArticle \
    -element MedlineCitation/PMID \
    -block PubDate -sep "-" -element Year,Month \
    -element Article/ArticleTitle \
    -block Abstract -element AbstractText | \
  awk 'BEGIN {FS="\t"; OFS=","} {print $1,$2,$3,$4,$5}' > \
  "$OUTPUT_DIR/summary.csv"
```

### 17. Literature Alert System

```bash
#!/bin/bash
# literature_alert.sh - Check for new publications

LAST_CHECK="2024-01-01"
QUERY="CRISPR therapeutics"

# Check for new papers since last check
new_count=$(esearch -db pubmed -query "$QUERY" | \
  efilter -mindate "$LAST_CHECK" | \
  xtract -pattern Count -element Count)

if [ "$new_count" -gt 0 ]; then
  echo "Found $new_count new publications!"
  esearch -db pubmed -query "$QUERY" | \
    efilter -mindate "$LAST_CHECK" | \
    efetch -format abstract | head -500
else
  echo "No new publications found."
fi
```

### 18. Grant Proposal References

```bash
# Gather references for specific methods
methods=("RNA-seq" "ChIP-seq" "ATAC-seq" "single-cell RNA-seq")
for method in "${methods[@]}"; do
  echo "=== $method ===" >> methods_references.txt
  esearch -db pubmed -query "$method [TIAB] AND 2023 [PDAT]" -retmax 10 | \
    efetch -format medline | \
    grep -E "^(TI|AB|PMID|SO):" >> methods_references.txt
  echo "" >> methods_references.txt
  sleep 1
done
```

## Educational Examples

### 19. Teaching PubMed Search Syntax

```bash
# Demonstrate field qualifiers
echo "=== Different search types ==="
echo "1. Title only:"
esearch -db pubmed -query "cancer [TITL]" -retmax 3 | efetch -format uid
sleep 1

echo "2. Title/Abstract:"
esearch -db pubmed -query "cancer [TIAB]" -retmax 3 | efetch -format uid
sleep 1

echo "3. MeSH terms:"
esearch -db pubmed -query "neoplasms [MESH]" -retmax 3 | efetch -format uid
```

### 20. Compare Search Strategies

```bash
# Compare different search approaches
strategies=(
  '"lung cancer"'
  'lung AND cancer'
  'lung cancer [TIAB]'
  '((lung OR pulmonary) AND (cancer OR carcinoma)) [TIAB]'
)

for strategy in "${strategies[@]}"; do
  count=$(esearch -db pubmed -query "$strategy" | \
    xtract -pattern Count -element Count)
  printf "%-50s %8d results\n" "$strategy" "$count"
  sleep 1
done
```

## Integration Examples

### 21. Combine with Zotero

```bash
# Export to CSV for Zotero import
esearch -db pubmed -query "your research topic" -retmax 200 | \
  efetch -format xml | \
  xtract -pattern PubmedArticle \
    -element MedlineCitation/PMID \
    -element Article/ArticleTitle \
    -block Author -position first -sep " " -element LastName,Initials \
    -element Journal/Title \
    -block PubDate -sep "/" -element Year,Month | \
  awk 'BEGIN {FS="\t"; print "PMID,Title,First Author,Journal,Year"} 
       {print $1","$2","$3","$4","$5}' > zotero_import.csv
```

### 22. Create Reading List

```bash
# Generate HTML reading list
echo "<html><body><h1>Literature Reading List</h1>" > reading_list.html

esearch -db pubmed -query "important topic" -retmax 20 | \
  efetch -format xml | \
  while read -r line; do
    if [[ "$line" =~ "<ArticleTitle>" ]]; then
      title=$(echo "$line" | sed 's/<[^>]*>//g')
      echo "<h2>$title</h2>" >> reading_list.html
    elif [[ "$line" =~ "<AbstractText>" ]]; then
      abstract=$(echo "$line" | sed 's/<[^>]*>//g')
      echo "<p>$abstract</p><hr>" >> reading_list.html
    fi
  done

echo "</body></html>" >> reading_list.html
```

### 23. Monitor Competitor Publications

```bash
#!/bin/bash
# monitor_competitors.sh

COMPANIES=("Moderna" "BioNTech" "Pfizer" "AstraZeneca")
DAYS_BACK=7

for company in "${COMPANIES[@]}"; do
  echo "=== $company publications (last $DAYS_BACK days) ==="
  esearch -db pubmed -query "$company [AFFL]" | \
    efilter -days "$DAYS_BACK" | \
    efetch -format abstract | head -300
  echo ""
  sleep 2
done
```

## Quick Reference Examples

### 24. One-Liners for Common Tasks

```bash
# Count papers by year
seq 2010 2023 | xargs -I {} bash -c 'echo -n "{}: "; esearch -db pubmed -query "cancer" | efilter -mindate {} -maxdate {} | xtract -pattern Count -element Count; sleep 0.5'

# Get PMIDs from search
esearch -db pubmed -query "your query" | efetch -format uid > pmids.txt

# Download abstracts in parallel
cat pmids.txt | xargs -P 5 -I {} bash -c 'efetch -db pubmed -id {} -format abstract > {}.txt; sleep 0.3'

# Create author frequency list
cat *.xml | xtract -pattern PubmedArticle -block Author -sep " " -element LastName,Initials | sort | uniq -c | sort -rn
```

### 25. Quality Control Checks

```bash
# Check for retracted articles
esearch -db pubmed -query "retracted publication [PTYP]" | \
  efilter -mindate 2022 | \
  efetch -format abstract | grep -i "retract" | head -10

# Verify data completeness
esearch -db pubmed -query "your query" -retmax 10 | \
  efetch -format xml | \
  xtract -pattern PubmedArticle \
    -element MedlineCitation/PMID \
    -if Abstract -exists -lbl "Has abstract" -else -lbl "No abstract"
```

## Tips for Using These Examples

1. **Modify queries** to match your research interests
2. **Adjust retmax values** based on your needs
3. **Add sleep commands** when processing large batches
4. **Save intermediate results** for debugging
5. **Combine examples** to create custom workflows
6. **Test with small samples** before full runs
7. **Check rate limits** and add API keys if needed
8. **Document your modifications** for reproducibility

path: INSTALL.md
content: # EDirect Installation and Configuration ⚠️

## ⚠️ Important Security Warning

**Before proceeding, understand that:**

1. **External script execution**: This installation guide involves downloading and executing official NCBI scripts
2. **System modifications**: Will modify your PATH environment variable
3. **Permission requirements**: May require system-level package installation

**Always follow these security practices:**
- Review script content after downloading
- Validate in a test environment
- Do not pipe remote scripts directly to shell
- Regularly update and verify

## Prerequisites

- Unix-like system (Linux, macOS, or Windows with Cygwin/WSL)
- Internet connection
- curl or wget installed

**Important:** This skill uses **local installation only** – no Docker or containers required. All tools run directly on your system.

**Security Advisory:** Always verify the source of any installation script. The official NCBI EDirect installer is hosted at `ftp.ncbi.nlm.nih.gov`. When using automated installation, download the script to a file first and review it if desired, rather than piping directly to your shell. This protects against potential supply-chain attacks or compromised mirrors.

## Installation Methods

### 🔴 Security First: User Confirmation Steps

**Before executing any installation command, you must:**
1. Understand the commands that will be executed
2. Confirm network connection security
3. Backup important data
4. Prepare rollback plans

### Method 1: Secure Manual Installation

```bash
# =========== Step 1: Download ===========
# Download to file, do not execute directly
wget -q https://ftp.ncbi.nlm.nih.gov/entrez/entrezdirect/install-edirect.sh

# =========== Step 2: Review ===========
# View script content (review is important)
cat install-edirect.sh | head -100  # View first 100 lines
# Or use a text editor to view complete content

# =========== Step 3: Analyze ===========
# Check what operations the script will perform
grep -n "wget\|curl\|chmod\|export" install-edirect.sh

# =========== Step 4: Execute ===========
# Execute manually only after review
chmod +x install-edirect.sh
./install-edirect.sh
```

### ⛔ Unsafe Practices (Prohibited)

**Never do this:**
```bash
# ❌ Dangerous: Piping remote script directly to shell
curl https://example.com/install.sh | bash

# ❌ Dangerous: Executing without review
sh -c "$(curl -fsSL https://example.com/install.sh)"
```

**Reasons:**
- Cannot review script content
- Cannot control mid-execution stops
- May be modified by man-in-the-middle attacks
- Cannot audit executed operations

### Method 2: Alternative Installation with Curl

If using curl, follow the same security steps:

```bash
# Secure way: Download, review, execute
curl -fsSL https://ftp.ncbi.nlm.nih.gov/entrez/entrezdirect/install-edirect.sh -o install-edirect.sh
less install-edirect.sh  # Review
chmod +x install-edirect.sh
./install-edirect.sh
```

This script will:
1. Download EDirect scripts and programs from the official NCBI repository
2. Create an `edirect` folder in your home directory
3. Add the directory to your PATH

**Security Note:** Avoid piping remote scripts directly to your shell (e.g., `sh -c "curl | bash"`). Downloading first allows you to verify the content before execution, following best security practices.

### Method 2: Manual Installation (Alternative)

If the automated script doesn't work, you can manually install:

```bash
# Create edirect directory
mkdir -p ~/edirect
cd ~/edirect

# Download and extract the package
curl -O https://ftp.ncbi.nlm.nih.gov/entrez/entrezdirect/edirect.tar.gz
tar -xzf edirect.tar.gz

# Add to PATH
echo 'export PATH="$HOME/edirect:$PATH"' >> ~/.bashrc
source ~/.bashrc
```

## Post-Installation Configuration

### 1. Configure API Key (For increased rate limits)

For increased rate limits (10 requests/second vs 3 requests/second), obtain an API key:

1. Create an NCBI account at https://www.ncbi.nlm.nih.gov/
2. Go to Account Settings → API Key Management
3. Generate a new API key

Add the API key to your environment:

```bash
# For current session
export NCBI_API_KEY=your_api_key_here

# For permanent configuration
echo 'export NCBI_API_KEY="your_api_key_here"' >> ~/.bashrc
echo 'export NCBI_API_KEY="your_api_key_here"' >> ~/.zshrc  # If using zsh
```

### 2. Configure Email Address (Optional - identifies you to NCBI)

NCBI requests that you identify yourself with an email address:

```bash
export NCBI_EMAIL="[REDACTED_SECRET]"
echo 'export NCBI_EMAIL="[REDACTED_SECRET]"' >> ~/.bashrc
```

### 3. Set Tool Configuration

EDirect tools can be configured with default options:

```bash
# Create configuration file if needed
touch ~/.ncbirc

# Set default format preferences
export EFETCH_FORMAT="xml"  # Default output format
```

## Verification

Test your installation:

```bash
# Check if commands are available
esearch -help
efetch -help

# Test a simple query
esearch -db pubmed -query "test" -retmax 1
```

Expected output should show help text without errors.

## Troubleshooting

### Common Issues

#### 1. "Command not found"
```bash
# Check if edirect is in PATH
echo $PATH | grep edirect

# If not, add it manually
export PATH="$HOME/edirect:$PATH"
```

#### 2. Permission Denied
```bash
# Make scripts executable
chmod +x ~/edirect/*.pl
chmod +x ~/edirect/*.pm
```

#### 3. Perl Module Errors
EDirect requires certain Perl modules. Install them:

```bash
# On Ubuntu/Debian
sudo apt-get install perl libwww-perl libxml-simple-perl

# On CentOS/RHEL
sudo yum install perl perl-libwww-perl perl-XML-Simple

# On macOS with Homebrew
brew install perl
cpan install LWP::Simple XML::Simple
```

#### 4. Proxy Configuration
If behind a proxy:

```bash
export ftp_proxy="http://proxy.example.com:8080"
export http_proxy="http://proxy.example.com:8080"
export https_proxy="http://proxy.example.com:8080"
```

#### 5. SSL Certificate Issues
If you encounter SSL certificate errors, update your system's CA certificates:

```bash
# Ubuntu/Debian
sudo apt-get update
sudo apt-get install ca-certificates

# CentOS/RHEL
sudo yum update ca-certificates

# macOS
brew install ca-certificates
```

Also ensure your system time is correct:
```bash
date  # check system time
# If incorrect, sync with NTP
sudo ntpdate pool.ntp.org  # or use timedatectl
```

**Important:** Never disable SSL verification by setting PERL_LWP_SSL_VERIFY_HOSTNAME=0 in production or when handling sensitive data. This weakens security and exposes you to man-in-the-middle attacks.

## Performance Tuning

### 1. Increase Timeouts for Large Queries
```bash
export EUTILS_TIMEOUT=60  # Increase timeout to 60 seconds
```

### 2. Enable Compression for Large Transfers
```bash
export EUTILS_COMPRESS=1  # Enable gzip compression
```

### 3. Cache Configuration
```bash
# Set cache directory
export NCBI_CACHE_DIR="$HOME/.ncbi/cache"
mkdir -p "$NCBI_CACHE_DIR"
```

## Uninstallation

To remove EDirect:

```bash
# Remove the edirect directory
rm -rf ~/edirect

# Remove from PATH in your shell configuration files
# Edit ~/.bashrc, ~/.bash_profile, ~/.zshrc, etc.
# Remove the line: export PATH="$HOME/edirect:$PATH"
```

## Platform-Specific Notes

### macOS
- Comes with Perl pre-installed
- May need to install command line tools: `xcode-select --install`

### Windows (WSL/Cygwin)
- Works well with Windows Subsystem for Linux (WSL)
- Cygwin requires Perl and related packages

### Linux Servers
- Most distributions have required Perl modules in repositories
- `screen` or `tmux` can be used for long-running queries

## Security Audit Checklist

### Pre-Installation Checks (Required)
- [ ] Confirm network environment security
- [ ] Backup important data
- [ ] Prepare rollback scripts
- [ ] Understand all commands to be executed

### Script Review (Required)
- [ ] Download script to local file
- [ ] View full script content (at least first 200 lines)
- [ ] Check all external URLs
- [ ] Understand commands the script will execute
- [ ] Confirm no hidden dangerous operations

### Environment Isolation Options
- [ ] Validate first in a test environment
- [ ] Use non-privileged user accounts
- [ ] Set up dedicated working directories
- [ ] Configure firewall rules

### Execution Monitoring
- [ ] Enable command logging
- [ ] Monitor network traffic
- [ ] Regularly check system integrity
- [ ] Set up anomaly alerts

## Containerized Installation Option

For production environments, containers can be used:

```bash
# Use Docker to run EDirect
docker run -it --rm \
  -v $(pwd)/data:/data \
  ncbi/edirect:latest \
  esearch -db pubmed -query "test"
```

## Compliance Notes

This installation process is suitable for:
- ✅ Personal research environments
- ✅ Academic institution laboratories
- ✅ Testing and development systems

Not suitable for:
- ⚠️ Production servers
- ⚠️ Critical business systems
- ⚠️ Multi-user shared environments
- ⚠️ Scenarios without audit capabilities

## Final Reminder

**You are fully responsible for this system installation.** Before proceeding, ensure you:
1. Understand all risks
2. Have sufficient technical capability
3. Have comprehensive backup and recovery plans
4. Accept potential security consequences

## Next Steps

Only proceed to the following after fully understanding the above security warnings:
- [BASICS.md](BASICS.md) Basic usage guide
- [EXAMPLES.md](EXAMPLES.md) Practical examples
- [ADVANCED.md](ADVANCED.md) Advanced techniques

path: OPENCLAW_INTEGRATION.md
content: # OpenClaw Integration Guide ⚠️

**Important Security Reminder**: You must complete secure installation steps (see [INSTALL.md](INSTALL.md)) before using this skill. This skill executes external commands and requires careful auditing.

## Secure Usage Guidelines

### ⚠️ Pre-Execution Checks
Before each use of the `exec` tool, ensure:
1. Understand the commands to be executed
2. Confirm EDirect is properly installed
3. Check network connection security
4. Set appropriate timeout periods

### Command Execution Audit
```bash
# Write complex commands to script files for auditability
# 1. Create script file
echo 'esearch -db pubmed -query "$1" | efetch -format abstract | head -200' > search.sh

# 2. Review script content
cat search.sh

# 3. Set permissions
chmod +x search.sh

# 4. Execute script
exec -c './search.sh "CRISPR"'
```

## Basic Usage in OpenClaw

### Running EDirect Commands

Use OpenClaw's `exec` tool to run EDirect commands:

```bash
# Simple search example
exec -c 'esearch -db pubmed -query "CRISPR [TIAB]" | efetch -format abstract | head -200'
```

### Using the Included Scripts

```bash
# Run the batch fetch script
exec -c 'cd /root/.openclaw/workspace/pubmed-edirect && ./scripts/batch_fetch_abstracts.sh'

# With custom parameters
exec -c 'cd /root/.openclaw/workspace/pubmed-edirect && ./scripts/search_export_csv.sh "machine learning" 50 ml_results.csv'
```

## Common OpenClaw Workflows

### 1. Literature Search and Summary

```bash
# Search and get a summary
exec -c '
  echo "Searching for recent review articles..."
  esearch -db pubmed -query "cancer immunotherapy" | \
    efilter -mindate 2023 -query "review [PTYP]" | \
    efetch -format abstract | head -500
'
```

### 2. Data Extraction to Workspace

```bash
# Extract data and save to workspace
exec -c '
  esearch -db pubmed -query "your topic" -retmax 100 | \
    efetch -format xml | \
    xtract -pattern PubmedArticle \
      -element PMID Title Year | \
    awk "BEGIN {FS=\"\\t\"; print \"PMID,Title,Year\"} {print \$1\",\"\$2\",\"\$3}" > /root/.openclaw/workspace/literature.csv
  echo "Data saved to workspace/literature.csv"
'
```

### 3. Automated Literature Monitoring

Create a cron job in OpenClaw to check for new publications:

```bash
# Check for new papers daily
exec -c '
  cd /root/.openclaw/workspace/pubmed-edirect
  ./scripts/publication_trends.sh "your research area" $(date +%Y -d "1 year ago") $(date +%Y) daily_trend.csv
  # Send notification if significant new publications
'
```

## OpenClaw-Specific Tips

### 1. Managing Large Outputs

```bash
# Pipe to less for long outputs
exec -c 'esearch -db pubmed -query "broad topic" | efetch -format abstract | less -N'

# Save to file and then read
exec -c '
  esearch -db pubmed -query "query" | efetch -format abstract > /tmp/results.txt
  head -100 /tmp/results.txt
'
```

### 2. Error Handling in OpenClaw

```bash
# Use error checking
exec -c '
  if esearch -db pubmed -query "test" > /dev/null 2>&1; then
    echo "EDirect is working correctly"
  else
    echo "Error: EDirect may not be installed or configured"
    echo "Please check INSTALL.md"
  fi
'
```

### 3. Interactive Search Sessions

```bash
# Create an interactive search function
exec -c '
  search_pubmed() {
    local query="$1"
    local max="${2:-20}"
    echo "Searching: $query"
    esearch -db pubmed -query "$query" -retmax "$max" | \
      efetch -format abstract | head -$((max * 10))
  }
  
  # Test it
  search_pubmed "COVID-19 vaccine" 10
'
```

## Integration with Other OpenClaw Skills

### Combine with Text Processing

```bash
# Extract and analyze text
exec -c '
  esearch -db pubmed -query "topic" -retmax 50 | \
    efetch -format abstract | \
    grep -i "keyword" | \
    wc -l | \
    xargs echo "Occurrences of keyword:"
'
```

### Use with File Management

```bash
# Organize results in workspace
exec -c '
  mkdir -p /root/.openclaw/workspace/literature/$(date +%Y-%m-%d)
  esearch -db pubmed -query "daily search" | \
    efetch -format xml > /root/.openclaw/workspace/literature/$(date +%Y-%m-%d)/results.xml
  echo "Results saved to workspace"
'
```

## Creating OpenClaw Commands

You can create aliases or functions in your OpenClaw environment:

```bash
# Add to your shell configuration
alias pubmed-search='esearch -db pubmed'
alias pubmed-fetch='efetch -db pubmed -format abstract'

# Or create functions
pmsearch() {
  esearch -db pubmed -query "$1" ${2:+-retmax $2} | \
    efetch -format abstract | head -${3:-200}
}
```

## Example: Complete Literature Review Workflow

```bash
# 1. Initial search
exec -c '
  echo "=== Initial Broad Search ==="
  esearch -db pubmed -query "main topic" | \
    efilter -mindate 2020 | \
    xtract -pattern Count -element Count
  echo " papers since 2020"
'

# 2. Refine search
exec -c '
  echo "=== Refined Search ==="
  esearch -db pubmed -query "specific aspect [TIAB]" | \
    efilter -query "review [PTYP]" | \
    efetch -format abstract | head -300
'

# 3. Extract data for analysis
exec -c '
  echo "=== Data Extraction ==="
  esearch -db pubmed -query "final query" -retmax 100 | \
    efetch -format xml | \
    xtract -pattern PubmedArticle \
      -element PMID Year Title | \
    sort -k2 > /root/.openclaw/workspace/extracted_data.tsv
  echo "Data extracted to workspace"
'
```

## Troubleshooting in OpenClaw

### Common Issues

1. **EDirect not found**
   ```bash
   exec -c 'export PATH="$HOME/edirect:$PATH" && esearch -help'
   ```

2. **Permission denied on scripts**
   ```bash
   exec -c 'chmod +x /root/.openclaw/workspace/pubmed-edirect/scripts/*.sh'
   ```

3. **Rate limiting errors**
   ```bash
   exec -c 'export NCBI_API_KEY="your_key" && esearch -db pubmed -query "test"'
   ```

### Testing Your Setup

```bash
# Run a complete test
exec -c '
  echo "Testing EDirect installation..."
  if command -v esearch >/dev/null 2>&1; then
    echo "✓ EDirect commands available"
    count=$(esearch -db pubmed -query "test" | xtract -pattern Count -element Count 2>/dev/null || echo "error")
    if [[ "$count" =~ ^[0-9]+$ ]]; then
      echo "✓ NCBI connection working ($count results for 'test')"
      echo "✓ Setup is complete and functional"
    else
      echo "✗ Could not connect to NCBI"
    fi
  else
    echo "✗ EDirect not in PATH"
    echo "Please check INSTALL.md"
  fi
'
```

## Best Practices for OpenClaw Use

1. **Use workspace directory** for storing results
2. **Add delays** in batch operations to respect rate limits
3. **Log your queries** for reproducibility
4. **Combine with other tools** (grep, awk, sort) for processing
5. **Create reusable scripts** for common workflows
6. **Monitor API usage** if using an API key
7. **Backup important results** from workspace

## Getting Help

- Review the skill documentation files
- Check EDirect official documentation
- Test queries in small batches first
- Use the `-help` option with any EDirect command

path: REFERENCE.md
content: # EDirect Quick Reference

## Command Reference

### Core Commands

| Command | Description | Basic Syntax |
|---------|-------------|--------------|
| `esearch` | Search databases | `esearch -db DATABASE -query "TERMS"` |
| `efetch` | Retrieve records | `efetch -db DATABASE -id ID -format FORMAT` |
| `elink` | Find related records | `elink -db DATABASE -id ID -target TARGET` |
| `efilter` | Filter results | `efilter -mindate DATE -maxdate DATE` |
| `xtract` | Extract XML data | `xtract -pattern PATTERN -element ELEMENT` |
| `einfo` | Database info | `einfo -db DATABASE` |
| `epost` | Post IDs to history | `epost -db DATABASE -id ID1,ID2` |

### Common Options

| Option | Description | Example |
|--------|-------------|---------|
| `-db` | Database name | `-db pubmed` |
| `-query` | Search terms | `-query "cancer [TIAB]"` |
| `-id` | Record IDs | `-id 1234567,2345678` |
| `-format` | Output format | `-format abstract` |
| `-retmax` | Max results | `-retmax 100` |
| `-retstart` | Start offset | `-retstart 100` |
| `-mindate` | Minimum date | `-mindate 2020` |
| `-maxdate` | Maximum date | `-maxdate 2023` |
| `-days` | Days back | `-days 30` |
| `-target` | Target database | `-target gene` |
| `-log` | Log to history | `-log` |

## PubMed Field Qualifiers

### Content Fields

| Qualifier | Field | Description |
|-----------|-------|-------------|
| `[TIAB]` | Title/Abstract | Words in title or abstract |
| `[TITL]` | Title | Words in title only |
| `[AB]` | Abstract | Words in abstract only |
| `[TW]` | Text Word | Words anywhere in citation |
| `[ALL]` | All Fields | All searchable fields |

### Author Fields

| Qualifier | Field | Description |
|-----------|-------|-------------|
| `[AUTH]` | Author | Any author |
| `[FAUT]` | First Author | First author only |
| `[LAUT]` | Last Author | Last author only |
| `[AU]` | Author | Alternative to [AUTH] |

### Journal Fields

| Qualifier | Field | Description |
|-----------|-------|-------------|
| `[JOUR]` | Journal | Journal name |
| `[TA]` | Journal Title Abbreviation | Abbreviated journal name |
| `[JT]` | Journal Title | Full journal name |

### Date Fields

| Qualifier | Field | Description |
|-----------|-------|-------------|
| `[PDAT]` | Publication Date | Date article published |
| `[EDAT]` | Entry Date | Date added to PubMed |
| `[MHDA]` | MeSH Date | Date MeSH terms added |

### MeSH Fields

| Qualifier | Field | Description |
|-----------|-------|-------------|
| `[MESH]` | MeSH Terms | Medical Subject Headings |
| `[MAJR]` | MeSH Major Topic | Major MeSH terms |
| `[SH]` | Subheadings | MeSH subheadings |
| `[MH]` | MeSH Terms | Alternative to [MESH] |

### Publication Types

| Qualifier | Field | Description |
|-----------|-------|-------------|
| `[PTYP]` | Publication Type | Type of article |
| `[PT]` | Publication Type | Alternative to [PTYP] |

### Other Fields

| Qualifier | Field | Description |
|-----------|-------|-------------|
| `[AFFL]` | Affiliation | Author affiliation |
| `[LANG]` | Language | Article language |
| `[FILT]` | Filter | PubMed filters |
| `[UID]` | PMID | PubMed ID number |

## Common Filters

### Language Filters

| Filter | Description |
|--------|-------------|
| `english [LANG]` | English language articles |
| `chinese [LANG]` | Chinese language articles |
| `german [LANG]` | German language articles |

### Species Filters

| Filter | Description |
|--------|-------------|
| `humans [MESH]` | Human studies |
| `animals [MESH]` | Animal studies |
| `mice [MESH]` | Mouse studies |

### Article Type Filters

| Filter | Description |
|--------|-------------|
| `review [PTYP]` | Review articles |
| `clinical trial [PTYP]` | Clinical trials |
| `randomized controlled trial [PTYP]` | RCTs |
| `meta-analysis [PTYP]` | Meta-analyses |

### Other Filters

| Filter | Description |
|--------|-------------|
| `has abstract [FILT]` | Articles with abstracts |
| `free full text [FILT]` | Free full text available |
| `medline [FILT]` | In MEDLINE database |

## Output Formats

### PubMed Formats

| Format | Description | Use Case |
|--------|-------------|----------|
| `abstract` | Human-readable abstract | Reading articles |
| `medline` | MEDLINE format | Reference managers |
| `xml` | Structured XML | Data extraction |
| `uid` | PMIDs only | Getting IDs |
| `docsum` | Document summary | Quick overview |
| `asn.1` | ASN.1 format | Specialized tools |
| `json` | JSON format | Web applications |

### Sequence Formats

| Format | Description | Database |
|--------|-------------|----------|
| `fasta` | FASTA sequence | nuccore, protein |
| `gb` | GenBank flatfile | nuccore |
| `gp` | GenPept flatfile | protein |
| `gff` | GFF3 format | nuccore |

### Special Formats

| Format | Description | Use Case |
|--------|-------------|----------|
| `native` | Native format | Database default |
| `acc` | Accession only | Getting accessions |
| `list` | ID list | Simple lists |

## Database Names

### Literature Databases

| Database | Description |
|----------|-------------|
| `pubmed` | Biomedical literature |
| `pmc` | PubMed Central full-text |
| `books` | NCBI Bookshelf |
| `nlmcatalog` | NLM Catalog |

### Sequence Databases

| Database | Description |
|----------|-------------|
| `nuccore` | Nucleotide sequences |
| `protein` | Protein sequences |
| `gene` | Gene records |
| `genome` | Genome assemblies |

### Specialized Databases

| Database | Description |
|----------|-------------|
| `mesh` | Medical Subject Headings |
| `taxonomy` | Taxonomy database |
| `snp` | Single Nucleotide Polymorphism |
| `structure` | Molecular structures |
| `clinvar` | Clinical variants |
| `gtr` | Genetic testing registry |
| `biosample` | BioSample database |
| `bioproject` | BioProject database |

## xtract Patterns for PubMed

### Common Patterns

| Pattern | Description | Example |
|---------|-------------|---------|
| `PubmedArticle` | Complete article | `-pattern PubmedArticle` |
| `DocumentSummary` | Summary format | `-pattern DocumentSummary` |
| `Count` | Result count | `-pattern Count` |

### PubMed XML Elements

| Element | Path | Description |
|---------|------|-------------|
| PMID | `MedlineCitation/PMID` | PubMed ID |
| Title | `Article/ArticleTitle` | Article title |
| Abstract | `Abstract/AbstractText` | Abstract text |
| Journal | `Journal/Title` | Journal name |
| Year | `PubDate/Year` | Publication year |
| Author | `AuthorList/Author` | Author information |
| MeSH | `MeshHeadingList/MeshHeading` | MeSH terms |

### xtract Commands

| Command | Description | Example |
|---------|-------------|---------|
| `-pattern` | Main pattern | `-pattern PubmedArticle` |
| `-element` | Extract element | `-element Title` |
| `-block` | Block pattern | `-block Author` |
| `-if` | Conditional | `-if Language -equals eng` |
| `-sep` | Separator | `-sep ", "` |
| `-tab` | Tab replacement | `-tab ","` |
| `-def` | Default value | `-def "N/A"` |

## Date Formats

EDirect accepts multiple date formats:

| Format | Example | Description |
|--------|---------|-------------|
| YYYY | `2023` | Year only |
| YYYY/MM | `2023/06` | Year and month |
| YYYY/MM/DD | `2023/06/15` | Full date |
| Relative | `-days 30` | Days ago |

## Rate Limits

| Configuration | Requests/Second | Daily Limit |
|---------------|----------------|-------------|
| No API key | 3 | 10,000 |
| With API key | 10 | Unlimited |

**Best Practices:**
- Always use API key for production use
- Add delays: `sleep 0.5` between requests
- Cache results when possible
- Use batch operations for many IDs

## Common Error Messages

| Error | Cause | Solution |
|-------|-------|----------|
| `Command not found` | EDirect not in PATH | Add `$HOME/edirect` to PATH |
| `403 Forbidden` | Rate limit exceeded | Add API key, add delays |
| `No results found` | Invalid query | Check field qualifiers |
| `XML parsing error` | Malformed XML | Check format, use `-format xml` |
| `Network error` | Connection issue | Check internet, retry |

## Boolean Operators

| Operator | Description | Example |
|----------|-------------|---------|
| `AND` | Both terms required | `cancer AND therapy` |
| `OR` | Either term | `lung OR breast` |
| `NOT` | Exclude term | `cancer NOT review` |
| `()` | Grouping | `(lung OR breast) AND cancer` |

## Special Characters

| Character | Usage | Notes |
|-----------|-------|-------|
| `*` | Truncation | `immun*` finds immune, immunity, etc. |
| `" "` | Phrase search | `"clinical trial"` exact phrase |
| `[ ]` | Field qualifier | `[TIAB]` title/abstract field |
| `~` | Proximity | `"machine learning"~10` within 10 words |

## Environment Variables

| Variable | Purpose | Example |
|----------|---------|---------|
| `NCBI_API_KEY` | API key for rate limits | `export NCBI_API_KEY="your_key"` |
| `NCBI_EMAIL` | Identify to NCBI | `export NCBI_EMAIL="[REDACTED_SECRET]"` |
| `PATH` | EDirect location | `export PATH="$HOME/edirect:$PATH"` |
| `EUTILS_TIMEOUT` | Request timeout | `export EUTILS_TIMEOUT=60` |

## Quick Examples Cheat Sheet

```bash
# Basic search
esearch -db pubmed -query "your topic" | efetch -format abstract

# Get PMIDs only
esearch -db pubmed -query "search terms" | efetch -format uid

# Recent articles
esearch -db pubmed -query "topic" | efilter -days 30 | efetch -format abstract

# Specific field search
esearch -db pubmed -query "author [AUTH] AND topic [TIAB]"

# Extract to CSV
esearch -db pubmed -query "topic" | efetch -format xml | \
  xtract -pattern PubmedArticle -tab "," -element PMID,Title,Year

# Find related articles
elink -db pubmed -id 1234567 -related | efetch -format abstract

# Cross-database link
esearch -db pubmed -query "gene name" | elink -target gene | efetch -format docsum
```

## Tips for Effective Searching

1. **Start broad, then narrow** - Begin with general terms, add filters
2. **Use field qualifiers** - Be specific about where to search
3. **Check MeSH terms** - Use controlled vocabulary when possible
4. **Review similar articles** - Use `elink -related` to find more
5. **Save your searches** - Use `-log` option for reproducibility
6. **Validate with small samples** - Test queries with `-retmax 5` first
7. **Combine with other tools** - Pipe to `grep`, `awk`, `sort` for analysis

## Related Resources

- [EDirect Official Documentation](https://www.ncbi.nlm.nih.gov/books/NBK179288/)
- [PubMed Help](https://pubmed.ncbi.nlm.nih.gov/help/)
- [NCBI E-utilities](https://www.ncbi.nlm.nih.gov/books/NBK25501/)
- [MeSH Database](https://www.ncbi.nlm.nih.gov/mesh/)

path: scripts/batch_fetch_abstracts.sh
content: #!/bin/bash
# batch_fetch_abstracts.sh
# Fetch abstracts for a list of PMIDs

set -e  # Exit on error

INPUT_FILE="${1:-pmids.txt}"
OUTPUT_DIR="${2:-abstracts}"
DELAY="${3:-0.5}"  # Delay between requests in seconds

# Check if input file exists
if [ ! -f "$INPUT_FILE" ]; then
    echo "Error: Input file '$INPUT_FILE' not found"
    echo "Usage: $0 [input_file] [output_dir] [delay_seconds]"
    echo "Default: pmids.txt abstracts 0.5"
    exit 1
fi

# Create output directory
mkdir -p "$OUTPUT_DIR"

# Count total PMIDs
total=$(wc -l < "$INPUT_FILE")
echo "Processing $total PMIDs from $INPUT_FILE"
echo "Output directory: $OUTPUT_DIR"
echo "Delay between requests: ${DELAY}s"

# Process each PMID
count=0
success=0
fail=0

while read pmid; do
    # Skip empty lines
    [ -z "$pmid" ] && continue
    
    count=$((count + 1))
    output_file="${OUTPUT_DIR}/${pmid}.txt"
    
    echo -n "[$count/$total] PMID $pmid: "
    
    # Fetch abstract
    if efetch -db pubmed -id "$pmid" -format abstract > "$output_file" 2>/dev/null; then
        # Check if we got actual content (not empty or error)
        if [ -s "$output_file" ] && ! grep -q "Error" "$output_file" 2>/dev/null; then
            echo "✓"
            success=$((success + 1))
        else
            echo "✗ (empty/error)"
            rm -f "$output_file"
            fail=$((fail + 1))
        fi
    else
        echo "✗ (fetch failed)"
        fail=$((fail + 1))
    fi
    
    # Delay to respect rate limits
    if [ "$count" -lt "$total" ]; then
        sleep "$DELAY"
    fi
    
done < "$INPUT_FILE"

# Summary
echo ""
echo "=== Summary ==="
echo "Total processed: $count"
echo "Successful: $success"
echo "Failed: $fail"
echo "Output in: $OUTPUT_DIR/"

# Create combined file
if [ "$success" -gt 0 ]; then
    cat "$OUTPUT_DIR"/*.txt > "${OUTPUT_DIR}_combined.txt" 2>/dev/null || true
    echo "Combined file: ${OUTPUT_DIR}_combined.txt"
fi

path: scripts/publication_trends.sh
content: #!/bin/bash
# publication_trends.sh
# Analyze publication trends over time

set -e

QUERY="$1"
START_YEAR="${2:-2000}"
END_YEAR="${3:-$(date +%Y)}"
OUTPUT_FILE="${4:-trends.csv}"

if [ -z "$QUERY" ]; then
    echo "Error: Search query required"
    echo "Usage: $0 \"search query\" [start_year] [end_year] [output_file]"
    echo "Example: $0 \"machine learning\" 2010 2023 ml_trends.csv"
    exit 1
fi

echo "Analyzing publication trends for: $QUERY"
echo "Years: $START_YEAR to $END_YEAR"
echo "Output: $OUTPUT_FILE"

# Create output file with header
echo "year,count,cumulative" > "$OUTPUT_FILE"

total_cumulative=0
years_processed=0

# Process each year
for year in $(seq "$START_YEAR" "$END_YEAR"); do
    echo -n "Processing $year... "
    
    # Count publications for this year
    count=$(esearch -db pubmed -query "$QUERY" | \
        efilter -mindate "$year" -maxdate "$year" | \
        xtract -pattern Count -element Count 2>/dev/null || echo "0")
    
    # Ensure count is numeric
    if [[ ! "$count" =~ ^[0-9]+$ ]]; then
        count=0
    fi
    
    total_cumulative=$((total_cumulative + count))
    years_processed=$((years_processed + 1))
    
    # Write to CSV
    echo "$year,$count,$total_cumulative" >> "$OUTPUT_FILE"
    
    echo "$count publications"
    
    # Delay to respect rate limits
    if [ "$year" -lt "$END_YEAR" ]; then
        sleep 1
    fi
done

echo ""
echo "=== Analysis Complete ==="
echo "Years analyzed: $years_processed"
echo "Total publications: $total_cumulative"
echo "Output file: $OUTPUT_FILE"

# Generate simple visualization
if command -v gnuplot >/dev/null 2>&1; then
    echo ""
    echo "Generating plot..."
    
    gnuplot << EOF
    set terminal dumb size 80,30
    set title "Publication Trend: $QUERY"
    set xlabel "Year"
    set ylabel "Publications"
    set xrange [$START_YEAR:$END_YEAR]
    set xtics 5
    set key off
    plot "$OUTPUT_FILE" using 1:2 with linespoints
EOF
else
    echo ""
    echo "=== Data Summary ==="
    echo "Year Range: $START_YEAR - $END_YEAR"
    echo "Peak year(s):"
    
    # Find peak years
    tail -n +2 "$OUTPUT_FILE" | sort -t, -k2,2nr | head -3 | \
        while IFS=, read year count cumulative; do
            echo "  $year: $count publications"
        done
    
    echo ""
    echo "Recent trend (last 5 years):"
    tail -6 "$OUTPUT_FILE" | head -5 | \
        while IFS=, read year count cumulative; do
            echo "  $year: $count publications"
        done
fi

# Calculate growth rate if we have enough data
if [ "$years_processed" -ge 2 ]; then
    first_year_count=$(awk -F, -v y="$START_YEAR" '$1==y {print $2}' "$OUTPUT_FILE")
    last_year_count=$(awk -F, -v y="$END_YEAR" '$1==y {print $2}' "$OUTPUT_FILE")
    
    if [ "$first_year_count" -gt 0 ]; then
        growth_rate=$(echo "scale=2; ($last_year_count - $first_year_count) / $first_year_count * 100" | bc 2>/dev/null || echo "N/A")
        echo ""
        echo "Growth from $START_YEAR to $END_YEAR: ${growth_rate}%"
    fi
fi

# Create alternative text-based visualization
echo ""
echo "=== Text Visualization ==="
max_count=$(tail -n +2 "$OUTPUT_FILE" | cut -d, -f2 | sort -nr | head -1)
scale=$((max_count / 50 + 1))

tail -n +2 "$OUTPUT_FILE" | \
    while IFS=, read year count cumulative; do
        bars=$((count / scale))
        printf "%4s: %5d |" "$year" "$count"
        for ((i=0; i<bars; i++)); do
            printf "█"
        done
        printf "\n"
    done | tail -20

Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)