Package: lightshield
Lightshield (LS)This tool provides a fully automated system to permanently mirror the official Riot APIs content
with your local system. It uses a number of microservices to split requests done to different endpoints
across separated processes and allows scaling each service according to your current data needs.
This is especially usefull in the early stages of setting up a data repository.The setup is meant for tasks using alot of match data while not wanting to set up their own data pipeline structure.
Lightshield does currently not pullallmatch details data but only a select subset (for details
check the SQL schemas in the postgres folder). Changes can be made easily by simply expanding/replacing the service.Lightshield is optimized to not repeat calls unecessarily. This comes at cost of having data added in a less than
real time fashion.What Lightshield doesnotdo wellDeliver up to date dataWhile it can get up to date data on user its not set up for this by defaultGather data on unranked playerBy default Lightshield pulls data through the league endpoint which requires a user to be ranked.
As such getting information over unranked player is not supported by default but can be manually added.Structure in ShortLightshield handles data through a distributable scalable network of triangular microservice structures.
All data collected is stored in a dedicated postgres database. Task handling and scalability is provided through a
buffering redis database.Each cluster of services is responsible for a single server and requires an additional proxy for rate limiting and
handling of the API key.Each step of data processing is stored inside the Postgres database from which a single manager service creates tasks.
Worker services then process the tasks, add data to the db and close the task.RequirementsLightshield runs on docker and can either be built through this repository or by pulling the imagesdirectly from DockerHub.SetupEnv variablesCopy and rename the included secrets_template.env into secrets.envI. NetworkInitialize the network used to bundle all services together and allow communication:dockernetworkcreatelightshieldThe name can be changed but has to be updated in the compose files as well.
If you are planning on running it through docker swarm use the appropriate network type.II. DatabaseSet up a postgres database either locally, in docker (attached to the network) or remotely. The services currently expect
no password verification as such using a remote postgres library should only be done if you can limit access through other means.DB Connection details can be configured through a secrets.env file (template file included).Lightshield requires the in the postgres/ folder listed tables to be set up in the specified database under schemas
corresponding to the server they will contain data for. E.g. pulling data for NA1 requires setting up a schemana1(lower case)
with all tables inside said schema as well as a userna1which will be used to pull data from said schema.III. Centralized ServicesServices included in thecompose-global.yamlfile are meant to be run centralized meaning they run server-agnostic (usually as a one-off).
Currently this refers to the proxy service as well as the buffer redis db.
Start the services by properly refering to the compose file:# Pull (specify a tag if needed, defaults to latest)TAG=latestdocker-compose-fcompose-global.yamlpull# Or builddocker-compose-fcompose-global.yamlbuild# And then rundocker-compose-fcompose-global.yamlup-dIV. Server Specific StructureFor docker-composeRun docker compose with default parameters. The services require the selected server to be passed into the container via
the environment variableSERVER. In addition make sure to use different project names either through-p [project name]or through the env variableCOMPOSE_PROJECT_NAME. This will stop multiple server setups from overwriting one another.# Build from sourcedocker-composebuild# Or pull from docker hub (specify a tag if needed, defaults to latest)TAG=latestdocker-composepull# Run either with the -p tagSERVER=EUW1docker-compose-plightshield_euw1up-d# Or env variableSERVER=EUW1COMPOSE_PROJECT_NAME=lightshield_euw1docker-composeup-dFor docker-swarmFollow the same guidelines explained for docker-compose. The images can either be built or pulled from DockerHub.SERVERstill needs to be passed into the service.
The individual project name is passed through the stack name.SERVER=EUW1dockerstackdeploy-ccompose-services.yamllightshield_euw1The compose file includes the base_image service which is just a unified default image for all other services. As such
this isn't an actual runnable service and would usually just immediatly exit when run in docker-compose. Swarm however tries
to rebuild and restart the service continuously as such you need to manually remove it from the stack to avoid that.FunctionalityServices are structured in general in a triangular form using the persistent postgres DB as source of truth,
a singular non-scalable manager to select new tasks and a scalable amount of microservices to work through those tasks.2 lists are used to buffer tasks:A list of non-selected tasks that are yet to be worked on.A list of selected tasks that are being or have been work onThe manager service looks at both lists to determine if it should add a new task to the first non-selected task list.
Each worker service pulls tasks from the first list and adds them to the secondary list. Tasks are never marked as done
as the tasks are by default no longer eligible once inserted in the DB, e.g. if a summonerId has to be pulled then the manager
will only select accounts without summonerId as tasks. Once the summonerId is added it will automatically be ineligible.
All tasks that are older thanTASK_BLOCKING(secrets.env) minutes in the secondary selected list are periodically removed by the manager
therefore making space for new tasks. As this is the only way tasks are being removed make sure to keep the limit just high enough
to assure that tasks currently being worked on are not removed while not having the queue overflow with already done tasks.ServicesLeague RankingUses the league-exp endpoint to crawl all ranked user in a continuous circle. This service has no manager and only requires a one-off.
via theUPDATE_INTERVAL=1variable in the compose file you can limit the delay between cycles in hours. By default after finishing
with challenger the service will wait 1 hour to restart on Iron games.Summoner IDUsing the summoner endpoint pulls the remaining IDs for each account that were not included in the league ranking response.
This is a one time tasks for each user but it will take alot of initial requests until all are done.Match HistoryPull and update the match history of user. Prioritizes user for which no calls have been made so far and, once all user have
had their history pulled user that have had the most soloQ matches played since their last update.
Use theMIN_MATCHES=20variable to limit how many new matches a player has to play to even be considered for an update.
Because for each match 10 match-history are updated one should consider keeping this number high as to not make 10 match history
calls per new match. Setting it to 10 or 20 means that for each match played on average only 1 or .5 calls have to be made.Match DetailsPull match details for a buffered match_id. This service pulls match details and adds a select number of attributes to the DB
(check the SQL files for more info). If more/less data is needed you have to update the service + db schema accordingly.
Matches are pulled newest to oldest and are not pulled  for matches older than defined throughDETAILS_CUTOFF(secrets.env).Match HistoryCurrently not implemented, WIP.Lightshield ToolsTools and Code-Sections of the Lightshield Framework that were better fit to be provided through dependency
rather then included in the main project.What currently doesn't work:The keys used to save data in Redis are not linked to the API key, as such multiple keys have to use
multiple Redis servers.Ratelimiter (WIP)Multi-Host async ratelimiting service. The clients each sync via a central redis server.Set up the proxy in an async context with redis connection details.fromlightshield.proxyimportProxyimportaiohttpasyncdefrun():p=Proxy()# Initiate the redis connector in async contextawaitp.init(host='localhost',port=5432)Make calls directly to one endpoint.
Provide the server the calls are run against and an identifier that helps you recognize which endpoint the requests are
run againstasyncwithaiohttp.ClientSession(headers={'X-Riot-Token':''})assession:zone=awaitp.get_endpoint(server='europe',zone='league-exp')forpageinrange(1,10):zone.request('https://euw1.api.riotgames.com/lol/league-exp/v4/entries/RANKED_SOLO_5x5/SILVER/I?page=%s'%page,session)Settings (WIP)The settings file contains a number of variables that are used across the project.
Variables can be set through:ENV > config.json > defaultfromlightshieldimportsettingsheaders={'X-Riot-Token':settings.API_KEY}