Repository: PokemonGoF/PokemonGo-Bot

Issue:
[Feature Request] Releasing all pokemon in bag except X best ones
### Short Description

Currently it is possible to release all pokemon of some type except N best ones based on some criteria:
`"release": {
          "any": {"keep_best_iv": N}
}`
Ability to always keep N best pokemon in bag would be useful. For example, player may want to keep 150 best CP and 50 best IV pokemon always in his bag and to release everyone else.
### Possible solution

Implement new "pokemon name" `all` which is executed after all other rules (first we discard pokemon based on specific rules, than in what remains we keep best 150 pokemon):
`"release": {
    "all": { "keep_best_custom": "iv, cp, hp_max", "amount": 150 }
}`

This may be a naive implementation which ignores application order:
1. In `_release_pokemon_get_groups` we always return one extra fake group with pokemon_id = 10000. This group containa all pokemons from other groups (just fav/fort-filtered `inventory.pokemons().all()`).
2. On line 28 in `transfer_pokemon.py` we replace getting pokemon name with `pokemon_name = Pokemons.name_for(pokemon_id) if pokemon_id > 0 else 'all'`
3. Everything else works as expected.

If we want application order to be correct, things are more difficult (I don't know bot well enough to propose good solution).  As far as I can see:
1. Refactor lines 29-87 (contents of `for` loop except the first line) in `transfer_pokemon.py` in a separate function `_release_pokemon_group(pokemon_list, pokemon_name)` to launch it later below the for loop with pokemon name equals to `'all'`.
2. After the end of `for` loop:
   `group = [p for p in inventory.pokemons().all() if p.in_fort is False and p.is_favourite is False]`.
3. Call `_release_pokemon_group(group, 'all')`.
4. Done.
### How it would help others

Well, first of all -- it would be quite nice feature on its own. Also it can solve problems like #4468 quite easily and it would prevent exceeding pokemon bag (which is possible with current configuration options) limits and some other nice features.

Keeping before-the-launch pokemon intact from transfer/evolution
### Short Description

Add an option to preserve any pokemon account had before launching bot. Pokemon caught before botting should be excluded from transfer, evolution and nicknaming.
### Possible solution
1. In `Pokemon` class (inventory.py) Introduce boolean variable `self.is_original` which should be set to True for all pokemon in inventory at bot's early start time. For all bot-captured pokemon set it to False (either by default or at capture time).
2. Later add boolean option to `TransferPokemon`, `NicknamePokemon` and `EvolvePokemon` tasks to optionally preserve pokemon with self.is_original set to True.
### How it would help others

A lot of people play game seriously -- they give names to their "pets", have 10 snorlax pokemon for different gyms and etc. They have their own plans for manually captured pokemon and that is why they often cry that bot acts destructively. That is why keeping X best bots is not a substitute for this. They would be happy to see those pet pokemon intact :)

_p.s.:_ wrapping this logic above `is_favorite` is not an option because it is used in-game by user and setting it in automatic manner would be really disrespectful. The same goes for asking user to fav their pokemon (they have their own understanding for using favs).



Discussion:

If you're going to run an account using any form of 3rd party app (like our bot), you will probably lose your account. We're just not beating detection methods yet. 

I don't think it's unreasonable to ask folks to favorite those they want to keep. 

Why not creating a rule to keep your 10 best snorlax ?

@mjmadsen this is not an option due to _p.s._ section: they have their own uses for favoriting and don't want to mix it with this. Also, if user have lots of subjectively useful pokemon -- they would do quite a work to mark them all even it doesn't conflict with their way of favoriting pokemon.

@Anakin5 because it is just an example :) If we keep 10 best pokemon of every family, their count will go far beyond bag size. I tried to figure out some numerical boundaries (X best, X% strongest, all with CP/IV/etc > X% than the max, keep X of family, etc) but always got complains. Talked to some players and realized that the best solution is kind of stupid in terms of algorithms -- just don't touch player's pokemon if they don't permit it to you :)

@LitRidl If you want to cheat, sometimes you might have to make concessions. Since the bot restarts often, or might be stopped manually and rebooted, you would keep having more and more pokemon being saved. So we would have to add some form of logging to note which pokemon were there the first time the bot was loaded on that particular account. That's a boat load of work to save someone from being unhappy about giving up their favorite rules. Plus if we got the bot to a point that it won't get banned, those favs will end up sucking by the time you get into late game unless you happened to find that perfect IV snorlax, with the best moveset and decent starting CP manually. 

I get where you are coming from, but my point is there is already a simple method to accomplish the goal. 

I didn't say keep 10 best of every family. I said keep 10 best snorlax.
I am not sure what the state of the master branch right now, but you can do that on dev branch.

@mjmadsen Well, honestly as a programmer I really like your point of view.
Anyway, are favorited bots safe from `PokemonEvolve` and `PokemonOptimizer`?

You actually spotted one extra problem: if player used bot, normally played for some time and again decided to cheat -- we will have to reset his old reserved pokemon and log them somewhere again.

How would you suggest to solve this problem by writing plugin, for example? Creating new task which should run only once and log pokemon list to some storage is Ok, but how can I modify existing logics of Transfer/Evolve/Nickname without altering their code?

Maybe those kind of problems should be solved by adding some kind of filter functions for transfer/evolve/etc? For example, if user wants to have some very custom filters (keep only pokemon with CP >= 0.7 \* of pokemon with max CP, keep only pokemon not listed in some storage and etc), he/she can write them in some special python file (with user-defined functions for use within config-file) or something like that.
By the way, it would also be useful for level-independent configs because user will be able to use relative CP/IV/DPS/etc values if they wanted.

Yeah, I think you'd need to edit their logic or find some means to remove specific pokemon from the pokemon inventory but still maintain pokebag count correctly. 

Yeah, all tasks should be leaving favorites alone. I personally haven't looked at optimizer's code, but I'm pretty sure it holds true. 

@mjmadsen and what about custom filters and relative values? Should I create new feature requests or are they no-way?

Sorry to come off as a pessimist right away. I'm sure if you put up a PR with some from of "no touching" system and modified the tasks that might mess with those pokemon, we could push it through. 

@LitRidl
I already have a change coming for relative values. Where you can keep your top 10% or something like that. But there is only very little usage of that feature and it doesn't answer your core question.

For now, the requirement is not clear to me. I would prefer you explain more what you are trying to achieve and then we can think of a solution.
Storing pokemon at startup is the solution to a problem, not the problem. So let's see if there is better solution to your problem.

@Anakin5 Oh, that's great (about relative values). By the way, will there be any way to filter them level-wise? I mean, below/above average level CP, below/above pokemon level, etc.?

So, the problem. Players often initially have some unsorted pokemon bag. Of course, they can go through it and manually star pokemon they don't want to be transferred. But thus we force them to think and make decisions (that's not lovely thing for most people) so that they postpones the whole process. Another problem is that often they have different level of game understanding: if we release pokemon with high initial CP and put instead another of the same type with 1.0 perfection, they may be displeased. The same with opposite way :)

That is why I currently see only 2 good solutions:
1. Star/favorite scheme, but it forces player to make decisions and thus delays the whole thing,
2. Just keep pokemon bag if there is enough space.

By the way, in regard to @mjmadsen : if all bot-captured pokemon are transferred immediately (excluding xp trash like weedle, caterpie and etc. because they should be immediately evolved and than also released), there is no problem with persistence. That makes sense for default because usually we don't want to use player's (potential) lucky eggs or good candies.

Let's not jump on the solution yet.
- "star Pokemon they don't want to be transfered": can we come up with a list of reason why someone would not want a Pokemon to be transfered ? It seems you are implying some players might want to keep Pokemon for a reason that go beyond any logic. I think there is a logic and that the bot can take the same decision.
- You can keep both the best cp and the best iv , and/or the best whatever. So it is already possible to ask the bot to keep multiple pokemon based on different criteria. The thing you cannot do now is keep the worst, but yeah...

What is level filter, below/above average level CP, below/above pokemon level ?

Well, actually you got me right -- as I said, some players are not very well educated about the game. Some of them (well, not the majority) love pokemons without logical reasons (for example, one player cried about extremely weak Vaporeon -- he said it was his first vaporeon, it was special). Also, some of them are collectors -- they just try to keep all pokemon they love in some way. The problem is that I can't a priori guess this special things.

About level filters: currently CP values in configs are mainly hard-coded which makes them level-dependent. For example, pokemons of level 15 have average CP higher that level 5 pokemons. I think it is quite versatile for configs: for example, we can keep only those pokemon of family X who has CP greater than 70% of current global max/avg CP among player's pokemon.
About levels -- well, pokemon have them as far as I know. It seems quite logical to add pokemon level to filter, but again -- it would still be hard-coded trainer level-dependent value.

- If a player has special affinity toward his Pokemon, on top of having to go to see a doctor ;), he might not want to run the bot at all. It is quite clear today that running the bot might get you banned. So don't risk it with an account you care a lot.
  For the rare cases where there is no logic a all, I think the favorite option is a good one because that exactly what it means, it is your favorite :)
- The Evolve task can keep one a each pokemon which is ideal for people who always want to keep at least one of each. There is no such thing right now in the Pokemon optimizer but it can be added indeed.
- I think percentage based criteria are more "dangerous" than a fix number.
  Because 10% does not mean that you want to keep 10% of the family. It means that you want to keep to top 10% of a range. for ex with iv, if the best iv 0.9, you will keep all Pokemon between 0.81 and 0.9 (that is the top 10% of the range 0-0.9). If you have 10 Pokemon within a family with iv 0.8 and you suddenly get a 0.9, all those 0.8 will be transfered. That is much more aggressive than saying you want to keep the top 2, which would have keep them all.

The opposite scenario is true as well, you might end up keeping much more Pokemon than what you are willing to, because they are all in the 10% range.

About the level, it is already represented by ncp (normalized cp). The good things about ncp compared to level is that 1) you know the range from 0% to 100%, 2) you can see it in game with the half circle bar over the head of the pokemon.
Sorting pokemon by ncp give the same result as sorting them by level.

@Anakin5 by the way, about keeping pokemon with best CP/IV and etc. As far as I know, `keep_*` filters in `release` section controls releasing duplicate pokemon.
Is there currently any way to say bot "I want to have 150 strongest by CP/IV/etc pokemon in my overall pokemon bag"? Because `"any": { "keep_best_cp": 150 }` will keep 150 best pokemon for each type, not for the whole bag.

@LitRidl
I don't know about the release task but the optimizer will rank Pokemon so that duplicates have same rank, so both will be kept.
For releasing/evolving considering the whole bag, it is doable. It is dangerous setting though.


Resolving patch:
diff --git a/pokemongo_bot/cell_workers/transfer_pokemon.py b/pokemongo_bot/cell_workers/transfer_pokemon.py
--- a/pokemongo_bot/cell_workers/transfer_pokemon.py
+++ b/pokemongo_bot/cell_workers/transfer_pokemon.py
@@ -8,83 +8,30 @@
 from pokemongo_bot.datastore import Datastore
 from operator import attrgetter
 
+
 class TransferPokemon(Datastore, BaseTask):
     SUPPORTED_TASK_API_VERSION = 1
 
     def __init__(self, bot, config):
         super(TransferPokemon, self).__init__(bot, config)
+
     def initialize(self):
         self.min_free_slot = self.config.get('min_free_slot', 5)
         self.transfer_wait_min = self.config.get('transfer_wait_min', 1)
         self.transfer_wait_max = self.config.get('transfer_wait_max', 4)
 
     def work(self):
-
         if not self._should_work():
             return
 
         pokemon_groups = self._release_pokemon_get_groups()
         for pokemon_id, group in pokemon_groups.iteritems():
             pokemon_name = Pokemons.name_for(pokemon_id)
-            keep_best, keep_best_cp, keep_best_iv = self._validate_keep_best_config(pokemon_name)
-            #TODO continue list possible criteria
-            keep_best_possible_criteria = ['cp','iv', 'iv_attack', 'iv_defense', 'iv_stamina', 'moveset.attack_perfection','moveset.defense_perfection','hp','hp_max']
-            keep_best_custom, keep_best_criteria, keep_amount = self._validate_keep_best_config_custom(pokemon_name, keep_best_possible_criteria)
-
-            best_pokemon_ids = set()
-            order_criteria = 'none'
-            if keep_best:
-                if keep_best_cp >= 1:
-                    cp_limit = keep_best_cp
-                    best_cp_pokemons = sorted(group, key=lambda x: (x.cp, x.iv), reverse=True)[:cp_limit]
-                    best_pokemon_ids = set(pokemon.unique_id for pokemon in best_cp_pokemons)
-                    order_criteria = 'cp'
-
-                if keep_best_iv >= 1:
-                    iv_limit = keep_best_iv
-                    best_iv_pokemons = sorted(group, key=lambda x: (x.iv, x.cp), reverse=True)[:iv_limit]
-                    best_pokemon_ids |= set(pokemon.unique_id for pokemon in best_iv_pokemons)
-                    if order_criteria == 'cp':
-                        order_criteria = 'cp and iv'
-                    else:
-                        order_criteria = 'iv'
-            elif keep_best_custom:
-                limit = keep_amount
-                keep_best_criteria = [str(keep_best_criteria[x]) for x in range(len(keep_best_criteria))] # not sure if the u of unicode will stay, so make it go away
-                best_pokemons = sorted(group, key=attrgetter(*keep_best_criteria), reverse=True)[:limit]
-                best_pokemon_ids = set(pokemon.unique_id for pokemon in best_pokemons)
-                order_criteria = ' then '.join(keep_best_criteria)
-
-            if keep_best or keep_best_custom:
-                # remove best pokemons from all pokemons array
-                all_pokemons = group
-                best_pokemons = []
-                for best_pokemon_id in best_pokemon_ids:
-                    for pokemon in all_pokemons:
-                        if best_pokemon_id == pokemon.unique_id:
-                            all_pokemons.remove(pokemon)
-                            best_pokemons.append(pokemon)
-
-                transfer_pokemons = [pokemon for pokemon in all_pokemons if self.should_release_pokemon(pokemon,True)]
-
-                if transfer_pokemons:
-                    if best_pokemons:
-                        self.emit_event(
-                            'keep_best_release',
-                            formatted="Keeping best {amount} {pokemon}, based on {criteria}",
-                            data={
-                                'amount': len(best_pokemons),
-                                'pokemon': pokemon_name,
-                                'criteria': order_criteria
-                            }
-                        )
-                    for pokemon in transfer_pokemons:
-                        self.release_pokemon(pokemon)
-            else:
-                group = sorted(group, key=lambda x: x.cp, reverse=True)
-                for pokemon in group:
-                    if self.should_release_pokemon(pokemon):
-                        self.release_pokemon(pokemon)
+            self._release_pokemon_worst_in_group(group, pokemon_name)
+
+        group = [p for p in inventory.pokemons().all()
+                 if p.in_fort is False and p.is_favorite is False]
+        self._release_pokemon_worst_in_group(group, 'all')
 
     def _should_work(self):
         return inventory.Pokemons.get_space_left() <= self.min_free_slot
@@ -104,19 +51,93 @@ def _release_pokemon_get_groups(self):
 
         return pokemon_groups
 
-    def should_release_pokemon(self, pokemon, keep_best_mode = False):
+    def _release_pokemon_worst_in_group(self, group, pokemon_name):
+        keep_best, keep_best_cp, keep_best_iv = self._validate_keep_best_config(
+            pokemon_name)
+        # TODO continue list possible criteria
+        keep_best_possible_criteria = ['cp', 'iv', 'iv_attack', 'iv_defense', 'iv_stamina',
+                                       'moveset.attack_perfection', 'moveset.defense_perfection', 'hp', 'hp_max']
+        keep_best_custom, keep_best_criteria, keep_amount = self._validate_keep_best_config_custom(
+            pokemon_name, keep_best_possible_criteria)
+
+        best_pokemon_ids = set()
+        order_criteria = 'none'
+        if keep_best:
+            if keep_best_cp >= 1:
+                cp_limit = keep_best_cp
+                best_cp_pokemons = sorted(group, key=lambda x: (
+                    x.cp, x.iv), reverse=True)[:cp_limit]
+                best_pokemon_ids = set(
+                    pokemon.unique_id for pokemon in best_cp_pokemons)
+                order_criteria = 'cp'
+
+            if keep_best_iv >= 1:
+                iv_limit = keep_best_iv
+                best_iv_pokemons = sorted(group, key=lambda x: (
+                    x.iv, x.cp), reverse=True)[:iv_limit]
+                best_pokemon_ids |= set(
+                    pokemon.unique_id for pokemon in best_iv_pokemons)
+                if order_criteria == 'cp':
+                    order_criteria = 'cp and iv'
+                else:
+                    order_criteria = 'iv'
+        elif keep_best_custom:
+            limit = keep_amount
+            # not sure if the u of unicode will stay, so make it go away
+            keep_best_criteria = [str(keep_best_criteria[x])
+                                  for x in range(len(keep_best_criteria))]
+            best_pokemons = sorted(group, key=attrgetter(
+                *keep_best_criteria), reverse=True)[:limit]
+            best_pokemon_ids = set(
+                pokemon.unique_id for pokemon in best_pokemons)
+            order_criteria = ' then '.join(keep_best_criteria)
+
+        if keep_best or keep_best_custom:
+            # remove best pokemons from all pokemons array
+            all_pokemons = group
+            best_pokemons = []
+            for best_pokemon_id in best_pokemon_ids:
+                for pokemon in all_pokemons:
+                    if best_pokemon_id == pokemon.unique_id:
+                        all_pokemons.remove(pokemon)
+                        best_pokemons.append(pokemon)
+
+            transfer_pokemons = [
+                pokemon for pokemon in all_pokemons if self.should_release_pokemon(pokemon, True)]
+
+            if transfer_pokemons:
+                if best_pokemons:
+                    self.emit_event(
+                        'keep_best_release',
+                        formatted="Keeping best {amount} {pokemon}, based on {criteria}",
+                        data={
+                            'amount': len(best_pokemons),
+                            'pokemon': pokemon_name,
+                            'criteria': order_criteria
+                        }
+                    )
+                for pokemon in transfer_pokemons:
+                    self.release_pokemon(pokemon)
+        else:
+            group = sorted(group, key=lambda x: x.cp, reverse=True)
+            for pokemon in group:
+                if self.should_release_pokemon(pokemon):
+                    self.release_pokemon(pokemon)
+
+    def should_release_pokemon(self, pokemon, keep_best_mode=False):
         release_config = self._get_release_config_for(pokemon.name)
 
         if (keep_best_mode
-            and not release_config.has_key('never_release')
-            and not release_config.has_key('always_release')
-            and not release_config.has_key('release_below_cp')
-            and not release_config.has_key('release_below_iv')):
+                and not release_config.has_key('never_release')
+                and not release_config.has_key('always_release')
+                and not release_config.has_key('release_below_cp')
+                and not release_config.has_key('release_below_iv')):
             return True
 
         cp_iv_logic = release_config.get('logic')
         if not cp_iv_logic:
-            cp_iv_logic = self._get_release_config_for('any').get('logic', 'and')
+            cp_iv_logic = self._get_release_config_for(
+                'any').get('logic', 'and')
 
         release_results = {
             'cp': False,
@@ -167,8 +188,10 @@ def release_pokemon(self, pokemon):
             if self.bot.config.test:
                 candy_awarded = 1
             else:
-                response_dict = self.bot.api.release_pokemon(pokemon_id=pokemon.unique_id)
-                candy_awarded = response_dict['responses']['RELEASE_POKEMON']['candy_awarded']
+                response_dict = self.bot.api.release_pokemon(
+                    pokemon_id=pokemon.unique_id)
+                candy_awarded = response_dict['responses'][
+                    'RELEASE_POKEMON']['candy_awarded']
         except KeyError:
             return
 
@@ -189,13 +212,15 @@ def release_pokemon(self, pokemon):
         )
         with self.bot.database as conn:
             c = conn.cursor()
-            c.execute("SELECT COUNT(name) FROM sqlite_master WHERE type='table' AND name='transfer_log'")
+            c.execute(
+                "SELECT COUNT(name) FROM sqlite_master WHERE type='table' AND name='transfer_log'")
 
         result = c.fetchone()
 
         while True:
             if result[0] == 1:
-                conn.execute('''INSERT INTO transfer_log (pokemon, iv, cp) VALUES (?, ?, ?)''', (pokemon.name, pokemon.iv, pokemon.cp))
+                conn.execute('''INSERT INTO transfer_log (pokemon, iv, cp) VALUES (?, ?, ?)''',
+                             (pokemon.name, pokemon.iv, pokemon.cp))
                 break
             else:
                 self.emit_event(


Test patch: